[==========] Running 25 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 21 tests from TxnParticipantITest
[ RUN ] TxnParticipantITest.TestReplicateParticipantOps
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 08:07:39.970788 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:33207
I20240503 08:07:39.971096 17235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 08:07:39.971251 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:39.975962 17235 server_base.cc:1034] running on GCE node
W20240503 08:07:39.976493 17241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:39.976912 17240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:39.977612 17243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:39.977855 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:39.977907 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:39.977941 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723659977939 us; error 0 us; skew 500 ppm
I20240503 08:07:39.982834 17235 webserver.cc:460] Webserver started at http://127.16.212.254:36065/ using document root <none> and password file <none>
I20240503 08:07:39.982990 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:39.983045 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:39.983144 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:39.984225 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c"
format_stamp: "Formatted at 2024-05-03 08:07:39 on dist-test-slave-zl2t"
I20240503 08:07:39.985728 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:07:39.986290 17248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:39.986471 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:39.986542 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c"
format_stamp: "Formatted at 2024-05-03 08:07:39 on dist-test-slave-zl2t"
I20240503 08:07:39.986627 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.014050 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:40.014323 17235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 08:07:40.014446 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:40.018255 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:33207
I20240503 08:07:40.018327 17300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:33207 every 8 connection(s)
I20240503 08:07:40.018965 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:40.021550 17301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c: Bootstrap starting.
I20240503 08:07:40.022321 17301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.022560 17301 log.cc:824] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c: Log is configured to *not* fsync() on all Append() calls
I20240503 08:07:40.023114 17301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c: No bootstrap required, opened a new log
I20240503 08:07:40.024423 17301 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" member_type: VOTER }
I20240503 08:07:40.024519 17301 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.024536 17301 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d4fab09ffe48fdbaf8085d0eb0d36c, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.024670 17301 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" member_type: VOTER }
I20240503 08:07:40.024729 17301 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:40.024749 17301 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:40.024796 17301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:40.025314 17301 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" member_type: VOTER }
I20240503 08:07:40.025406 17301 leader_election.cc:302] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6d4fab09ffe48fdbaf8085d0eb0d36c; no voters:
I20240503 08:07:40.025552 17301 leader_election.cc:288] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:40.025594 17304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:40.025734 17304 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 1 LEADER]: Becoming Leader. State: Replica: b6d4fab09ffe48fdbaf8085d0eb0d36c, State: Running, Role: LEADER
I20240503 08:07:40.025760 17301 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:40.025872 17304 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" member_type: VOTER }
I20240503 08:07:40.026237 17306 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d4fab09ffe48fdbaf8085d0eb0d36c. Latest consensus state: current_term: 1 leader_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" member_type: VOTER } }
I20240503 08:07:40.026214 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d4fab09ffe48fdbaf8085d0eb0d36c" member_type: VOTER } }
I20240503 08:07:40.026383 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:40.026369 17306 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:40.026816 17313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:40.027256 17313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:40.027400 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:07:40.028482 17313 catalog_manager.cc:1347] Generated new cluster ID: 4fab57d88580488e8989efd9f4651a17
I20240503 08:07:40.028545 17313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:40.037827 17313 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:07:40.038372 17313 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:40.050781 17313 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c: Generated new TSK 0
I20240503 08:07:40.050990 17313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:07:40.059392 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:40.061794 17323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.062244 17324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.062803 17326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.063539 17235 server_base.cc:1034] running on GCE node
I20240503 08:07:40.063668 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.063707 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.063725 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723660063726 us; error 0 us; skew 500 ppm
I20240503 08:07:40.064479 17235 webserver.cc:460] Webserver started at http://127.16.212.193:37937/ using document root <none> and password file <none>
I20240503 08:07:40.064576 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.064628 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.064675 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.065004 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "13955f9f2fb348c19da448138bd5c0af"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-zl2t"
I20240503 08:07:40.066125 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:40.066711 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.066835 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.066893 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "13955f9f2fb348c19da448138bd5c0af"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-zl2t"
I20240503 08:07:40.066946 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.100914 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:40.101195 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:40.101413 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:40.102025 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:40.102072 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.102110 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:40.102136 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.106333 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:42835
I20240503 08:07:40.106561 17394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:42835 every 8 connection(s)
I20240503 08:07:40.106750 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:07:40.109117 17235 server_base.cc:1034] running on GCE node
W20240503 08:07:40.109748 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.110301 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.110824 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.111055 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.111104 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.111125 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723660111125 us; error 0 us; skew 500 ppm
I20240503 08:07:40.111807 17235 webserver.cc:460] Webserver started at http://127.16.212.194:40091/ using document root <none> and password file <none>
I20240503 08:07:40.111897 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.111937 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.111974 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.112140 17395 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33207
I20240503 08:07:40.112224 17395 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:40.112298 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "cac6319c221445f68ba54fc3c5804866"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-zl2t"
I20240503 08:07:40.112387 17395 heartbeater.cc:511] Master 127.16.212.254:33207 requested a full tablet report, sending...
I20240503 08:07:40.112982 17265 ts_manager.cc:194] Registered new tserver with Master: 13955f9f2fb348c19da448138bd5c0af (127.16.212.193:42835)
I20240503 08:07:40.113332 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:40.113875 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.113924 17265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36528
I20240503 08:07:40.114023 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.114059 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "cac6319c221445f68ba54fc3c5804866"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-zl2t"
I20240503 08:07:40.114092 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.121570 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:40.121769 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:40.121907 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:40.122088 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:40.122115 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.122151 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:40.122177 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.125427 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:41395
I20240503 08:07:40.125489 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:41395 every 8 connection(s)
I20240503 08:07:40.125872 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:40.127759 17474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.128500 17475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:40.129101 17477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:40.129643 17235 server_base.cc:1034] running on GCE node
I20240503 08:07:40.129761 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:40.129796 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:40.129822 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723660129821 us; error 0 us; skew 500 ppm
I20240503 08:07:40.130097 17471 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33207
I20240503 08:07:40.130162 17471 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:40.130265 17471 heartbeater.cc:511] Master 127.16.212.254:33207 requested a full tablet report, sending...
I20240503 08:07:40.130502 17265 ts_manager.cc:194] Registered new tserver with Master: cac6319c221445f68ba54fc3c5804866 (127.16.212.194:41395)
I20240503 08:07:40.130561 17235 webserver.cc:460] Webserver started at http://127.16.212.195:45831/ using document root <none> and password file <none>
I20240503 08:07:40.130659 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:40.130703 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:40.130765 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:40.130985 17265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36538
I20240503 08:07:40.131036 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "7ce7f8daffe2461aa4698d4da2ff2d20"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-zl2t"
I20240503 08:07:40.131933 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:40.132395 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.132544 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.132586 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "7ce7f8daffe2461aa4698d4da2ff2d20"
format_stamp: "Formatted at 2024-05-03 08:07:40 on dist-test-slave-zl2t"
I20240503 08:07:40.132637 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:40.174109 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:40.174377 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:40.174563 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:40.174798 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:40.174826 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.174865 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:40.174891 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.178359 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:41251
I20240503 08:07:40.178440 17545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:41251 every 8 connection(s)
I20240503 08:07:40.182042 17546 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33207
I20240503 08:07:40.182106 17546 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:40.182211 17546 heartbeater.cc:511] Master 127.16.212.254:33207 requested a full tablet report, sending...
I20240503 08:07:40.182456 17265 ts_manager.cc:194] Registered new tserver with Master: 7ce7f8daffe2461aa4698d4da2ff2d20 (127.16.212.195:41251)
I20240503 08:07:40.182965 17265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36548
I20240503 08:07:40.182965 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004470403s
I20240503 08:07:40.183429 17235 test_util.cc:268] Using random seed: 916928387
I20240503 08:07:40.190752 17265 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36556:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:07:40.191010 17265 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:07:40.196774 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 115c3a727f8b4bd68baf9e19ca4d3530 (DEFAULT_TABLE table=test-workload [id=da150c44b8e543088505282e8814bc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:07:40.196942 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115c3a727f8b4bd68baf9e19ca4d3530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:40.196774 17359 tablet_service.cc:1467] Processing CreateTablet for tablet 115c3a727f8b4bd68baf9e19ca4d3530 (DEFAULT_TABLE table=test-workload [id=da150c44b8e543088505282e8814bc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:07:40.197168 17359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115c3a727f8b4bd68baf9e19ca4d3530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:40.198110 17561 tablet_bootstrap.cc:492] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20: Bootstrap starting.
I20240503 08:07:40.198385 17435 tablet_service.cc:1467] Processing CreateTablet for tablet 115c3a727f8b4bd68baf9e19ca4d3530 (DEFAULT_TABLE table=test-workload [id=da150c44b8e543088505282e8814bc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:07:40.198487 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115c3a727f8b4bd68baf9e19ca4d3530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:40.198570 17561 tablet_bootstrap.cc:654] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.198724 17562 tablet_bootstrap.cc:492] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af: Bootstrap starting.
I20240503 08:07:40.199059 17562 tablet_bootstrap.cc:654] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.199537 17562 tablet_bootstrap.cc:492] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af: No bootstrap required, opened a new log
I20240503 08:07:40.199558 17561 tablet_bootstrap.cc:492] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20: No bootstrap required, opened a new log
I20240503 08:07:40.199592 17562 ts_tablet_manager.cc:1367] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:40.199594 17561 ts_tablet_manager.cc:1367] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:40.199762 17565 tablet_bootstrap.cc:492] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: Bootstrap starting.
I20240503 08:07:40.199743 17562 raft_consensus.cc:348] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.199847 17562 raft_consensus.cc:374] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.199877 17562 raft_consensus.cc:729] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13955f9f2fb348c19da448138bd5c0af, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.199928 17562 consensus_queue.cc:260] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.199755 17561 raft_consensus.cc:348] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.200030 17561 raft_consensus.cc:374] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.200058 17561 raft_consensus.cc:729] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce7f8daffe2461aa4698d4da2ff2d20, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.200090 17562 ts_tablet_manager.cc:1398] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:40.200109 17561 consensus_queue.cc:260] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.200170 17565 tablet_bootstrap.cc:654] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:40.200218 17395 heartbeater.cc:503] Master 127.16.212.254:33207 was elected leader, sending a full tablet report...
I20240503 08:07:40.200245 17561 ts_tablet_manager.cc:1398] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:40.200337 17546 heartbeater.cc:503] Master 127.16.212.254:33207 was elected leader, sending a full tablet report...
I20240503 08:07:40.200745 17565 tablet_bootstrap.cc:492] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: No bootstrap required, opened a new log
I20240503 08:07:40.200788 17565 ts_tablet_manager.cc:1367] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:40.200927 17565 raft_consensus.cc:348] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.201027 17565 raft_consensus.cc:374] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:40.201053 17565 raft_consensus.cc:729] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac6319c221445f68ba54fc3c5804866, State: Initialized, Role: FOLLOWER
I20240503 08:07:40.201126 17565 consensus_queue.cc:260] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.201310 17565 ts_tablet_manager.cc:1398] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:07:40.201426 17471 heartbeater.cc:503] Master 127.16.212.254:33207 was elected leader, sending a full tablet report...
I20240503 08:07:40.202235 17569 raft_consensus.cc:482] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:40.202291 17569 raft_consensus.cc:504] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.202505 17569 leader_election.cc:288] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13955f9f2fb348c19da448138bd5c0af (127.16.212.193:42835), 7ce7f8daffe2461aa4698d4da2ff2d20 (127.16.212.195:41251)
I20240503 08:07:40.205422 17369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530" candidate_uuid: "cac6319c221445f68ba54fc3c5804866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13955f9f2fb348c19da448138bd5c0af" is_pre_election: true
I20240503 08:07:40.205523 17369 raft_consensus.cc:2454] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cac6319c221445f68ba54fc3c5804866 in term 0.
I20240503 08:07:40.205533 17520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530" candidate_uuid: "cac6319c221445f68ba54fc3c5804866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" is_pre_election: true
I20240503 08:07:40.205600 17520 raft_consensus.cc:2454] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cac6319c221445f68ba54fc3c5804866 in term 0.
I20240503 08:07:40.205734 17409 leader_election.cc:302] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce7f8daffe2461aa4698d4da2ff2d20, cac6319c221445f68ba54fc3c5804866; no voters:
I20240503 08:07:40.205823 17569 raft_consensus.cc:2781] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:40.205878 17569 raft_consensus.cc:482] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:40.205896 17569 raft_consensus.cc:3037] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:40.206605 17569 raft_consensus.cc:504] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.206766 17569 leader_election.cc:288] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [CANDIDATE]: Term 1 election: Requested vote from peers 13955f9f2fb348c19da448138bd5c0af (127.16.212.193:42835), 7ce7f8daffe2461aa4698d4da2ff2d20 (127.16.212.195:41251)
I20240503 08:07:40.206879 17369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530" candidate_uuid: "cac6319c221445f68ba54fc3c5804866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13955f9f2fb348c19da448138bd5c0af"
I20240503 08:07:40.206929 17369 raft_consensus.cc:3037] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:40.206916 17520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530" candidate_uuid: "cac6319c221445f68ba54fc3c5804866" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20"
I20240503 08:07:40.207008 17520 raft_consensus.cc:3037] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:40.207515 17520 raft_consensus.cc:2454] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cac6319c221445f68ba54fc3c5804866 in term 1.
I20240503 08:07:40.207535 17369 raft_consensus.cc:2454] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cac6319c221445f68ba54fc3c5804866 in term 1.
I20240503 08:07:40.207706 17408 leader_election.cc:302] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13955f9f2fb348c19da448138bd5c0af, cac6319c221445f68ba54fc3c5804866; no voters:
I20240503 08:07:40.207803 17569 raft_consensus.cc:2781] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:40.207908 17569 raft_consensus.cc:686] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 LEADER]: Becoming Leader. State: Replica: cac6319c221445f68ba54fc3c5804866, State: Running, Role: LEADER
I20240503 08:07:40.207948 17569 consensus_queue.cc:237] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:40.208479 17265 catalog_manager.cc:5484] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 reported cstate change: term changed from 0 to 1, leader changed from <none> to cac6319c221445f68ba54fc3c5804866 (127.16.212.194). New cstate: current_term: 1 leader_uuid: "cac6319c221445f68ba54fc3c5804866" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } health_report { overall_health: HEALTHY } } }
I20240503 08:07:40.222209 17369 raft_consensus.cc:1261] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 1 FOLLOWER]: Refusing update from remote peer cac6319c221445f68ba54fc3c5804866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:40.222218 17520 raft_consensus.cc:1261] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 1 FOLLOWER]: Refusing update from remote peer cac6319c221445f68ba54fc3c5804866: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:40.222635 17569 consensus_queue.cc:1035] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:40.222736 17571 consensus_queue.cc:1035] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:40.224584 17580 mvcc.cc:205] Tried to move back new op lower bound from 7023508112268460032 to 7023508112212361216. Current Snapshot: MvccSnapshot[applied={T|T < 7023508112268460032}]
I20240503 08:07:50.764139 17615 raft_consensus.cc:979] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: : Instructing follower 13955f9f2fb348c19da448138bd5c0af to start an election
I20240503 08:07:50.764227 17615 raft_consensus.cc:1067] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 LEADER]: Signalling peer 13955f9f2fb348c19da448138bd5c0af to start an election
I20240503 08:07:50.764431 17369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530"
dest_uuid: "13955f9f2fb348c19da448138bd5c0af"
from {username='slave'} at 127.0.0.1:47020
I20240503 08:07:50.764537 17369 raft_consensus.cc:482] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:07:50.764568 17369 raft_consensus.cc:3037] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:50.765270 17369 raft_consensus.cc:504] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:50.765440 17369 leader_election.cc:288] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [CANDIDATE]: Term 2 election: Requested vote from peers 7ce7f8daffe2461aa4698d4da2ff2d20 (127.16.212.195:41251), cac6319c221445f68ba54fc3c5804866 (127.16.212.194:41395)
I20240503 08:07:50.769147 17519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530" candidate_uuid: "13955f9f2fb348c19da448138bd5c0af" candidate_term: 2 candidate_status { last_received { term: 1 index: 13381 } } ignore_live_leader: true dest_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20"
I20240503 08:07:50.769222 17519 raft_consensus.cc:3037] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:50.769806 17519 raft_consensus.cc:2454] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13955f9f2fb348c19da448138bd5c0af in term 2.
I20240503 08:07:50.769922 17445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c3a727f8b4bd68baf9e19ca4d3530" candidate_uuid: "13955f9f2fb348c19da448138bd5c0af" candidate_term: 2 candidate_status { last_received { term: 1 index: 13381 } } ignore_live_leader: true dest_uuid: "cac6319c221445f68ba54fc3c5804866"
I20240503 08:07:50.769989 17445 raft_consensus.cc:3032] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:07:50.770010 17333 leader_election.cc:302] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13955f9f2fb348c19da448138bd5c0af, 7ce7f8daffe2461aa4698d4da2ff2d20; no voters:
I20240503 08:07:50.770013 17445 raft_consensus.cc:729] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cac6319c221445f68ba54fc3c5804866, State: Running, Role: LEADER
I20240503 08:07:50.770148 17445 consensus_queue.cc:260] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13381, Committed index: 13381, Last appended: 1.13381, Last appended by leader: 13381, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:50.770249 17621 raft_consensus.cc:2781] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:07:50.770260 17445 raft_consensus.cc:3037] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:07:50.770401 17621 raft_consensus.cc:686] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 2 LEADER]: Becoming Leader. State: Replica: 13955f9f2fb348c19da448138bd5c0af, State: Running, Role: LEADER
I20240503 08:07:50.770462 17621 consensus_queue.cc:237] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13381, Committed index: 13381, Last appended: 1.13381, Last appended by leader: 13381, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } }
I20240503 08:07:50.771052 17445 raft_consensus.cc:2454] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13955f9f2fb348c19da448138bd5c0af in term 2.
I20240503 08:07:50.771028 17256 catalog_manager.cc:5484] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af reported cstate change: term changed from 1 to 2, leader changed from cac6319c221445f68ba54fc3c5804866 (127.16.212.194) to 13955f9f2fb348c19da448138bd5c0af (127.16.212.193). New cstate: current_term: 2 leader_uuid: "13955f9f2fb348c19da448138bd5c0af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13955f9f2fb348c19da448138bd5c0af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:50.774755 17445 raft_consensus.cc:1261] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 2 FOLLOWER]: Refusing update from remote peer 13955f9f2fb348c19da448138bd5c0af: Log matching property violated. Preceding OpId in replica: term: 1 index: 13381. Preceding OpId from leader: term: 2 index: 13383. (index mismatch)
I20240503 08:07:50.774755 17519 raft_consensus.cc:1261] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 2 FOLLOWER]: Refusing update from remote peer 13955f9f2fb348c19da448138bd5c0af: Log matching property violated. Preceding OpId in replica: term: 1 index: 13381. Preceding OpId from leader: term: 2 index: 13383. (index mismatch)
I20240503 08:07:50.774979 17621 consensus_queue.cc:1035] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce7f8daffe2461aa4698d4da2ff2d20" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13382, Last known committed idx: 13381, Time since last communication: 0.000s
I20240503 08:07:50.775089 17622 consensus_queue.cc:1035] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [LEADER]: Connected to new peer: Peer: permanent_uuid: "cac6319c221445f68ba54fc3c5804866" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 41395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13382, Last known committed idx: 13381, Time since last communication: 0.000s
I20240503 08:07:50.781713 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:07:50.785058 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.785171 17235 tablet_replica.cc:330] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af: stopping tablet replica
I20240503 08:07:50.785256 17235 raft_consensus.cc:2229] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.785323 17235 raft_consensus.cc:2258] T 115c3a727f8b4bd68baf9e19ca4d3530 P 13955f9f2fb348c19da448138bd5c0af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.800562 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:07:50.801555 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:07:50.804759 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.804877 17235 tablet_replica.cc:330] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866: stopping tablet replica
I20240503 08:07:50.804944 17235 raft_consensus.cc:2229] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.805001 17235 raft_consensus.cc:2258] T 115c3a727f8b4bd68baf9e19ca4d3530 P cac6319c221445f68ba54fc3c5804866 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.826092 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:07:50.827103 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:07:50.830189 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:07:50.830318 17235 tablet_replica.cc:330] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20: stopping tablet replica
I20240503 08:07:50.830371 17235 raft_consensus.cc:2229] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:07:50.830425 17235 raft_consensus.cc:2258] T 115c3a727f8b4bd68baf9e19ca4d3530 P 7ce7f8daffe2461aa4698d4da2ff2d20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.843716 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:07:50.844619 17235 master.cc:537] Master@127.16.212.254:33207 shutting down...
I20240503 08:07:50.847551 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:07:50.847613 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:07:50.847636 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6d4fab09ffe48fdbaf8085d0eb0d36c: stopping tablet replica
I20240503 08:07:50.859321 17235 master.cc:552] Master@127.16.212.254:33207 shutdown complete.
[ OK ] TxnParticipantITest.TestReplicateParticipantOps (10906 ms)
[ RUN ] TxnParticipantITest.TestWaitOnFinalizeCommit
I20240503 08:07:50.877593 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:35831
I20240503 08:07:50.877736 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.879896 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.880509 17235 server_base.cc:1034] running on GCE node
W20240503 08:07:50.881014 17660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.881757 17659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.881911 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.881955 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.881969 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723670881969 us; error 0 us; skew 500 ppm
I20240503 08:07:50.882591 17235 webserver.cc:460] Webserver started at http://127.16.212.254:35641/ using document root <none> and password file <none>
I20240503 08:07:50.882700 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.882745 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.882790 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.883069 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "70169ad769f34f08960a6e46129527fa"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.883920 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.884343 17667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.884485 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.884536 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "70169ad769f34f08960a6e46129527fa"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.884572 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.895306 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.895475 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.897867 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:35831
I20240503 08:07:50.902258 17720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:50.903977 17719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:35831 every 8 connection(s)
I20240503 08:07:50.904104 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa: Bootstrap starting.
I20240503 08:07:50.906055 17720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:50.907018 17720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa: No bootstrap required, opened a new log
I20240503 08:07:50.907147 17720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70169ad769f34f08960a6e46129527fa" member_type: VOTER }
I20240503 08:07:50.907208 17720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:50.907230 17720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70169ad769f34f08960a6e46129527fa, State: Initialized, Role: FOLLOWER
I20240503 08:07:50.907289 17720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70169ad769f34f08960a6e46129527fa" member_type: VOTER }
I20240503 08:07:50.907330 17720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:07:50.907351 17720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:07:50.907379 17720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:50.907886 17720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70169ad769f34f08960a6e46129527fa" member_type: VOTER }
I20240503 08:07:50.907948 17720 leader_election.cc:302] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70169ad769f34f08960a6e46129527fa; no voters:
I20240503 08:07:50.908068 17720 leader_election.cc:288] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:07:50.908171 17723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:50.908210 17720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:07:50.908346 17723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 1 LEADER]: Becoming Leader. State: Replica: 70169ad769f34f08960a6e46129527fa, State: Running, Role: LEADER
I20240503 08:07:50.908406 17723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70169ad769f34f08960a6e46129527fa" member_type: VOTER }
I20240503 08:07:50.908636 17724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70169ad769f34f08960a6e46129527fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70169ad769f34f08960a6e46129527fa" member_type: VOTER } }
I20240503 08:07:50.908651 17725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70169ad769f34f08960a6e46129527fa. Latest consensus state: current_term: 1 leader_uuid: "70169ad769f34f08960a6e46129527fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70169ad769f34f08960a6e46129527fa" member_type: VOTER } }
I20240503 08:07:50.908759 17725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:50.908747 17724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [sys.catalog]: This master's current role is: LEADER
I20240503 08:07:50.909060 17733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:07:50.909193 17733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:07:50.909547 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:07:50.909691 17733 catalog_manager.cc:1347] Generated new cluster ID: 9a97a6e38c35466b9a5cab66aa0353f6
I20240503 08:07:50.909731 17733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:07:50.915139 17733 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:07:50.915570 17733 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:07:50.919952 17733 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa: Generated new TSK 0
I20240503 08:07:50.920025 17733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:07:50.925133 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.927131 17742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.927670 17744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.928442 17741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.929172 17235 server_base.cc:1034] running on GCE node
I20240503 08:07:50.929291 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.929323 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.929348 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723670929348 us; error 0 us; skew 500 ppm
I20240503 08:07:50.930094 17235 webserver.cc:460] Webserver started at http://127.16.212.193:43195/ using document root <none> and password file <none>
I20240503 08:07:50.930192 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.930243 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.930296 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.930586 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "251b0c376e494c1e8e36235bfec52c28"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.931555 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.932039 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.932199 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.932240 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "251b0c376e494c1e8e36235bfec52c28"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.932289 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.942836 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.943064 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.943195 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:50.943384 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:50.943410 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.943444 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:50.943470 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.946053 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:45823
I20240503 08:07:50.946450 17812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:45823 every 8 connection(s)
I20240503 08:07:50.946667 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.948783 17817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.949579 17820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.950086 17813 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35831
I20240503 08:07:50.950110 17235 server_base.cc:1034] running on GCE node
I20240503 08:07:50.950171 17813 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.950294 17813 heartbeater.cc:511] Master 127.16.212.254:35831 requested a full tablet report, sending...
I20240503 08:07:50.950577 17684 ts_manager.cc:194] Registered new tserver with Master: 251b0c376e494c1e8e36235bfec52c28 (127.16.212.193:45823)
W20240503 08:07:50.950666 17818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.950901 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:07:50.950938 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.951007 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723670951006 us; error 0 us; skew 500 ppm
I20240503 08:07:50.951323 17684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34854
I20240503 08:07:50.951750 17235 webserver.cc:460] Webserver started at http://127.16.212.194:43687/ using document root <none> and password file <none>
I20240503 08:07:50.951829 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.951922 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.951961 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.952262 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "40afc19265304286b877256f69d67bc6"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.953274 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:50.953861 17825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.953990 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:07:50.954051 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "40afc19265304286b877256f69d67bc6"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.954106 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.963979 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.964212 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.964362 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:50.964541 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:50.964568 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.964591 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:50.964618 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.967525 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:33223
I20240503 08:07:50.967592 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:33223 every 8 connection(s)
I20240503 08:07:50.967847 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:07:50.969693 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.970275 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:07:50.970814 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:07:50.970875 17889 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35831
I20240503 08:07:50.970940 17889 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.971073 17889 heartbeater.cc:511] Master 127.16.212.254:35831 requested a full tablet report, sending...
I20240503 08:07:50.971310 17684 ts_manager.cc:194] Registered new tserver with Master: 40afc19265304286b877256f69d67bc6 (127.16.212.194:33223)
I20240503 08:07:50.971607 17235 server_base.cc:1034] running on GCE node
I20240503 08:07:50.971735 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:07:50.971750 17684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34870
W20240503 08:07:50.971763 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:07:50.971911 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723670971910 us; error 0 us; skew 500 ppm
I20240503 08:07:50.972640 17235 webserver.cc:460] Webserver started at http://127.16.212.195:36693/ using document root <none> and password file <none>
I20240503 08:07:50.972713 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:07:50.972754 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:07:50.972790 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:07:50.973016 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "b2fa47c054b44925929f0f2528d33db8"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.973829 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:50.974228 17900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.974357 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.974406 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "b2fa47c054b44925929f0f2528d33db8"
format_stamp: "Formatted at 2024-05-03 08:07:50 on dist-test-slave-zl2t"
I20240503 08:07:50.974447 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:07:50.990499 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:07:50.990732 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:07:50.990870 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:07:50.991056 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:07:50.991092 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.991113 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:07:50.991139 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:07:50.994148 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:46581
I20240503 08:07:50.994223 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:46581 every 8 connection(s)
I20240503 08:07:50.997148 17964 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35831
I20240503 08:07:50.997215 17964 heartbeater.cc:464] Registering TS with master...
I20240503 08:07:50.997319 17964 heartbeater.cc:511] Master 127.16.212.254:35831 requested a full tablet report, sending...
I20240503 08:07:50.997551 17684 ts_manager.cc:194] Registered new tserver with Master: b2fa47c054b44925929f0f2528d33db8 (127.16.212.195:46581)
I20240503 08:07:50.997654 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003327683s
I20240503 08:07:50.998044 17684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34874
I20240503 08:07:50.998055 17235 test_util.cc:268] Using random seed: 927743014
I20240503 08:07:51.002791 17684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34882:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:07:51.002926 17684 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:07:51.007812 17928 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdd780345e14c418d9b12f6bd8c0152 (DEFAULT_TABLE table=test-workload [id=509f299c843e468dba37691aa7e5c415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:07:51.007917 17853 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdd780345e14c418d9b12f6bd8c0152 (DEFAULT_TABLE table=test-workload [id=509f299c843e468dba37691aa7e5c415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:07:51.007957 17777 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdd780345e14c418d9b12f6bd8c0152 (DEFAULT_TABLE table=test-workload [id=509f299c843e468dba37691aa7e5c415]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:07:51.007959 17928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdd780345e14c418d9b12f6bd8c0152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:51.008045 17853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdd780345e14c418d9b12f6bd8c0152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:51.009310 17979 tablet_bootstrap.cc:492] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: Bootstrap starting.
I20240503 08:07:51.008045 17777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdd780345e14c418d9b12f6bd8c0152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:07:51.009737 17979 tablet_bootstrap.cc:654] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:51.010285 17979 tablet_bootstrap.cc:492] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: No bootstrap required, opened a new log
I20240503 08:07:51.010327 17979 ts_tablet_manager.cc:1367] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:51.010489 17979 raft_consensus.cc:348] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.010524 17982 tablet_bootstrap.cc:492] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28: Bootstrap starting.
I20240503 08:07:51.010586 17979 raft_consensus.cc:374] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:51.010679 17979 raft_consensus.cc:729] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fa47c054b44925929f0f2528d33db8, State: Initialized, Role: FOLLOWER
I20240503 08:07:51.010738 17979 consensus_queue.cc:260] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.009445 17980 tablet_bootstrap.cc:492] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: Bootstrap starting.
I20240503 08:07:51.010936 17979 ts_tablet_manager.cc:1398] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:07:51.010948 17964 heartbeater.cc:503] Master 127.16.212.254:35831 was elected leader, sending a full tablet report...
I20240503 08:07:51.011073 17982 tablet_bootstrap.cc:654] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:51.011166 17980 tablet_bootstrap.cc:654] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: Neither blocks nor log segments found. Creating new log.
I20240503 08:07:51.011691 17982 tablet_bootstrap.cc:492] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28: No bootstrap required, opened a new log
I20240503 08:07:51.011698 17980 tablet_bootstrap.cc:492] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: No bootstrap required, opened a new log
I20240503 08:07:51.011772 17982 ts_tablet_manager.cc:1367] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:51.011775 17980 ts_tablet_manager.cc:1367] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:07:51.011935 17980 raft_consensus.cc:348] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.011934 17982 raft_consensus.cc:348] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.012030 17980 raft_consensus.cc:374] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:51.012065 17980 raft_consensus.cc:729] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40afc19265304286b877256f69d67bc6, State: Initialized, Role: FOLLOWER
I20240503 08:07:51.012039 17982 raft_consensus.cc:374] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:07:51.012138 17982 raft_consensus.cc:729] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251b0c376e494c1e8e36235bfec52c28, State: Initialized, Role: FOLLOWER
I20240503 08:07:51.012166 17980 consensus_queue.cc:260] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.012204 17982 consensus_queue.cc:260] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.012338 17980 ts_tablet_manager.cc:1398] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:07:51.012383 17889 heartbeater.cc:503] Master 127.16.212.254:35831 was elected leader, sending a full tablet report...
I20240503 08:07:51.012434 17813 heartbeater.cc:503] Master 127.16.212.254:35831 was elected leader, sending a full tablet report...
I20240503 08:07:51.012346 17982 ts_tablet_manager.cc:1398] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:07:51.026051 17983 raft_consensus.cc:482] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:07:51.026144 17983 raft_consensus.cc:504] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.026355 17983 leader_election.cc:288] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40afc19265304286b877256f69d67bc6 (127.16.212.194:33223), 251b0c376e494c1e8e36235bfec52c28 (127.16.212.193:45823)
I20240503 08:07:51.029660 17863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152" candidate_uuid: "b2fa47c054b44925929f0f2528d33db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40afc19265304286b877256f69d67bc6" is_pre_election: true
I20240503 08:07:51.029753 17863 raft_consensus.cc:2454] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2fa47c054b44925929f0f2528d33db8 in term 0.
I20240503 08:07:51.029676 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152" candidate_uuid: "b2fa47c054b44925929f0f2528d33db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251b0c376e494c1e8e36235bfec52c28" is_pre_election: true
I20240503 08:07:51.029798 17787 raft_consensus.cc:2454] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2fa47c054b44925929f0f2528d33db8 in term 0.
I20240503 08:07:51.029892 17901 leader_election.cc:302] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40afc19265304286b877256f69d67bc6, b2fa47c054b44925929f0f2528d33db8; no voters:
I20240503 08:07:51.029974 17983 raft_consensus.cc:2781] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:07:51.030012 17983 raft_consensus.cc:482] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:07:51.030043 17983 raft_consensus.cc:3037] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:51.030697 17983 raft_consensus.cc:504] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.030823 17983 leader_election.cc:288] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [CANDIDATE]: Term 1 election: Requested vote from peers 40afc19265304286b877256f69d67bc6 (127.16.212.194:33223), 251b0c376e494c1e8e36235bfec52c28 (127.16.212.193:45823)
I20240503 08:07:51.030943 17863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152" candidate_uuid: "b2fa47c054b44925929f0f2528d33db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40afc19265304286b877256f69d67bc6"
I20240503 08:07:51.030995 17863 raft_consensus.cc:3037] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:51.031015 17787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152" candidate_uuid: "b2fa47c054b44925929f0f2528d33db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251b0c376e494c1e8e36235bfec52c28"
I20240503 08:07:51.031066 17787 raft_consensus.cc:3037] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:07:51.031706 17787 raft_consensus.cc:2454] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fa47c054b44925929f0f2528d33db8 in term 1.
I20240503 08:07:51.031706 17863 raft_consensus.cc:2454] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fa47c054b44925929f0f2528d33db8 in term 1.
I20240503 08:07:51.031924 17904 leader_election.cc:302] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251b0c376e494c1e8e36235bfec52c28, b2fa47c054b44925929f0f2528d33db8; no voters:
I20240503 08:07:51.032011 17983 raft_consensus.cc:2781] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:07:51.032156 17983 raft_consensus.cc:686] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 LEADER]: Becoming Leader. State: Replica: b2fa47c054b44925929f0f2528d33db8, State: Running, Role: LEADER
I20240503 08:07:51.032202 17983 consensus_queue.cc:237] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:07:51.032806 17683 catalog_manager.cc:5484] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2fa47c054b44925929f0f2528d33db8 (127.16.212.195). New cstate: current_term: 1 leader_uuid: "b2fa47c054b44925929f0f2528d33db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } health_report { overall_health: UNKNOWN } } }
I20240503 08:07:51.051512 17787 raft_consensus.cc:1261] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 1 FOLLOWER]: Refusing update from remote peer b2fa47c054b44925929f0f2528d33db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:51.051544 17863 raft_consensus.cc:1261] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 1 FOLLOWER]: Refusing update from remote peer b2fa47c054b44925929f0f2528d33db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:07:51.051738 17989 consensus_queue.cc:1035] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:07:51.051860 17983 consensus_queue.cc:1035] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:07:56.358131 18000 log.cc:924] Time spent T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240503 08:08:01.568914 18009 raft_consensus.cc:979] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: : Instructing follower 251b0c376e494c1e8e36235bfec52c28 to start an election
I20240503 08:08:01.568995 18009 raft_consensus.cc:1067] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 LEADER]: Signalling peer 251b0c376e494c1e8e36235bfec52c28 to start an election
I20240503 08:08:01.569260 17786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152"
dest_uuid: "251b0c376e494c1e8e36235bfec52c28"
from {username='slave'} at 127.0.0.1:55290
I20240503 08:08:01.569366 17786 raft_consensus.cc:482] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:08:01.569396 17786 raft_consensus.cc:3037] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:01.571843 17786 raft_consensus.cc:504] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:08:01.572041 17786 leader_election.cc:288] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [CANDIDATE]: Term 2 election: Requested vote from peers 40afc19265304286b877256f69d67bc6 (127.16.212.194:33223), b2fa47c054b44925929f0f2528d33db8 (127.16.212.195:46581)
I20240503 08:08:01.574644 17862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152" candidate_uuid: "251b0c376e494c1e8e36235bfec52c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 13032 } } ignore_live_leader: true dest_uuid: "40afc19265304286b877256f69d67bc6"
I20240503 08:08:01.574733 17862 raft_consensus.cc:3037] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:01.574720 17938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdd780345e14c418d9b12f6bd8c0152" candidate_uuid: "251b0c376e494c1e8e36235bfec52c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 13032 } } ignore_live_leader: true dest_uuid: "b2fa47c054b44925929f0f2528d33db8"
I20240503 08:08:01.574813 17938 raft_consensus.cc:3032] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:08:01.574848 17938 raft_consensus.cc:729] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2fa47c054b44925929f0f2528d33db8, State: Running, Role: LEADER
I20240503 08:08:01.574909 17938 consensus_queue.cc:260] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13032, Committed index: 13032, Last appended: 1.13032, Last appended by leader: 13032, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:08:01.574999 17938 raft_consensus.cc:3037] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:01.575443 17862 raft_consensus.cc:2454] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 251b0c376e494c1e8e36235bfec52c28 in term 2.
I20240503 08:08:01.575634 17750 leader_election.cc:302] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251b0c376e494c1e8e36235bfec52c28, 40afc19265304286b877256f69d67bc6; no voters:
I20240503 08:08:01.575672 17938 raft_consensus.cc:2454] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 251b0c376e494c1e8e36235bfec52c28 in term 2.
I20240503 08:08:01.575805 18037 raft_consensus.cc:2781] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:08:01.575914 18037 raft_consensus.cc:686] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 2 LEADER]: Becoming Leader. State: Replica: 251b0c376e494c1e8e36235bfec52c28, State: Running, Role: LEADER
I20240503 08:08:01.575973 18037 consensus_queue.cc:237] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13032, Committed index: 13032, Last appended: 1.13032, Last appended by leader: 13032, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } }
I20240503 08:08:01.576440 17676 catalog_manager.cc:5484] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 reported cstate change: term changed from 1 to 2, leader changed from b2fa47c054b44925929f0f2528d33db8 (127.16.212.195) to 251b0c376e494c1e8e36235bfec52c28 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "251b0c376e494c1e8e36235bfec52c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "251b0c376e494c1e8e36235bfec52c28" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45823 } health_report { overall_health: HEALTHY } } }
I20240503 08:08:01.582867 17938 raft_consensus.cc:1261] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 2 FOLLOWER]: Refusing update from remote peer 251b0c376e494c1e8e36235bfec52c28: Log matching property violated. Preceding OpId in replica: term: 1 index: 13032. Preceding OpId from leader: term: 2 index: 13034. (index mismatch)
I20240503 08:08:01.582873 17862 raft_consensus.cc:1261] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 2 FOLLOWER]: Refusing update from remote peer 251b0c376e494c1e8e36235bfec52c28: Log matching property violated. Preceding OpId in replica: term: 1 index: 13032. Preceding OpId from leader: term: 2 index: 13034. (index mismatch)
I20240503 08:08:01.583067 18037 consensus_queue.cc:1035] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40afc19265304286b877256f69d67bc6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13033, Last known committed idx: 13032, Time since last communication: 0.000s
I20240503 08:08:01.583175 18038 consensus_queue.cc:1035] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2fa47c054b44925929f0f2528d33db8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13033, Last known committed idx: 13032, Time since last communication: 0.000s
I20240503 08:08:03.640434 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:08:03.644212 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:03.644352 17235 tablet_replica.cc:330] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28: stopping tablet replica
I20240503 08:08:03.644421 17235 raft_consensus.cc:2229] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:08:03.644500 17235 raft_consensus.cc:2258] T 4fdd780345e14c418d9b12f6bd8c0152 P 251b0c376e494c1e8e36235bfec52c28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:03.662530 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:08:03.663652 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:08:03.667469 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:03.667625 17235 tablet_replica.cc:330] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6: stopping tablet replica
I20240503 08:08:03.667677 17235 raft_consensus.cc:2229] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:03.667735 17235 raft_consensus.cc:2258] T 4fdd780345e14c418d9b12f6bd8c0152 P 40afc19265304286b877256f69d67bc6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:03.681838 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:08:03.682865 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:08:03.687418 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:03.687533 17235 tablet_replica.cc:330] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8: stopping tablet replica
I20240503 08:08:03.687599 17235 raft_consensus.cc:2229] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:03.687650 17235 raft_consensus.cc:2258] T 4fdd780345e14c418d9b12f6bd8c0152 P b2fa47c054b44925929f0f2528d33db8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:03.713424 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:08:03.714397 17235 master.cc:537] Master@127.16.212.254:35831 shutting down...
I20240503 08:08:03.717011 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:03.717078 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:03.717096 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 70169ad769f34f08960a6e46129527fa: stopping tablet replica
I20240503 08:08:03.729287 17235 master.cc:552] Master@127.16.212.254:35831 shutdown complete.
[ OK ] TxnParticipantITest.TestWaitOnFinalizeCommit (12871 ms)
[ RUN ] TxnParticipantITest.TestWaitOnAbortCommit
I20240503 08:08:03.749280 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:35761
I20240503 08:08:03.749464 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:03.751893 18077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:03.752815 18080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.753728 17235 server_base.cc:1034] running on GCE node
W20240503 08:08:03.754318 18078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.754479 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:03.754518 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:03.754534 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723683754534 us; error 0 us; skew 500 ppm
I20240503 08:08:03.755415 17235 webserver.cc:460] Webserver started at http://127.16.212.254:41745/ using document root <none> and password file <none>
I20240503 08:08:03.755506 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:03.755560 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:03.755609 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:03.755906 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "0725cbf7d12445139df44d6bff7b29b9"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.756913 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:03.757450 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.757650 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:08:03.757699 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "0725cbf7d12445139df44d6bff7b29b9"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.757753 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:03.765749 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:03.765965 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:03.768707 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:35761
I20240503 08:08:03.773432 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:35761 every 8 connection(s)
I20240503 08:08:03.773558 18138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:03.774595 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9: Bootstrap starting.
I20240503 08:08:03.774941 18138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:03.775594 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9: No bootstrap required, opened a new log
I20240503 08:08:03.775729 18138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0725cbf7d12445139df44d6bff7b29b9" member_type: VOTER }
I20240503 08:08:03.775785 18138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:03.775810 18138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725cbf7d12445139df44d6bff7b29b9, State: Initialized, Role: FOLLOWER
I20240503 08:08:03.775866 18138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0725cbf7d12445139df44d6bff7b29b9" member_type: VOTER }
I20240503 08:08:03.775898 18138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:03.775921 18138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:03.775954 18138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:03.776459 18138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0725cbf7d12445139df44d6bff7b29b9" member_type: VOTER }
I20240503 08:08:03.776517 18138 leader_election.cc:302] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0725cbf7d12445139df44d6bff7b29b9; no voters:
I20240503 08:08:03.776620 18138 leader_election.cc:288] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:03.776670 18141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:03.776772 18138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:03.776793 18141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0725cbf7d12445139df44d6bff7b29b9, State: Running, Role: LEADER
I20240503 08:08:03.776834 18141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0725cbf7d12445139df44d6bff7b29b9" member_type: VOTER }
I20240503 08:08:03.777005 18141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0725cbf7d12445139df44d6bff7b29b9. Latest consensus state: current_term: 1 leader_uuid: "0725cbf7d12445139df44d6bff7b29b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0725cbf7d12445139df44d6bff7b29b9" member_type: VOTER } }
I20240503 08:08:03.777017 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0725cbf7d12445139df44d6bff7b29b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0725cbf7d12445139df44d6bff7b29b9" member_type: VOTER } }
I20240503 08:08:03.777158 18141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:03.777240 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:03.777487 18148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:03.777640 18148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:03.778136 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:03.778265 18148 catalog_manager.cc:1347] Generated new cluster ID: 9dc9b27f868d428383b896382d25b4d6
I20240503 08:08:03.778304 18148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:03.784706 18148 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:03.785168 18148 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:03.791765 18148 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9: Generated new TSK 0
I20240503 08:08:03.791841 18148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:03.793730 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:03.796028 18162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:03.796859 18160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.797495 17235 server_base.cc:1034] running on GCE node
W20240503 08:08:03.798043 18159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.798185 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:03.798220 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:03.798233 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723683798233 us; error 0 us; skew 500 ppm
I20240503 08:08:03.798911 17235 webserver.cc:460] Webserver started at http://127.16.212.193:39659/ using document root <none> and password file <none>
I20240503 08:08:03.798985 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:03.799021 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:03.799072 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:03.799317 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "14236800ea834aa1895d2cf102699a82"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.800263 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:03.800706 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.800835 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.800891 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "14236800ea834aa1895d2cf102699a82"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.800926 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:03.820302 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:03.820474 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:03.820605 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:03.820812 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:03.820840 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.820863 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:03.820890 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.823555 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:45303
I20240503 08:08:03.823685 18230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:45303 every 8 connection(s)
I20240503 08:08:03.823957 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:03.826071 18235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:03.826903 18238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.827181 18231 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35761
I20240503 08:08:03.827241 18231 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:03.827324 18231 heartbeater.cc:511] Master 127.16.212.254:35761 requested a full tablet report, sending...
W20240503 08:08:03.827574 18236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.827606 18102 ts_manager.cc:194] Registered new tserver with Master: 14236800ea834aa1895d2cf102699a82 (127.16.212.193:45303)
I20240503 08:08:03.828163 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:03.828210 18102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59864
I20240503 08:08:03.828284 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:03.828320 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:03.828336 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723683828337 us; error 0 us; skew 500 ppm
I20240503 08:08:03.828951 17235 webserver.cc:460] Webserver started at http://127.16.212.194:34103/ using document root <none> and password file <none>
I20240503 08:08:03.829028 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:03.829084 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:03.829113 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:03.829398 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "3493dff7b1504ed290781c4f9cea9f74"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.830300 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:03.830834 18243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.830963 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.831013 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "3493dff7b1504ed290781c4f9cea9f74"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.831054 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:03.844079 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:03.844278 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:03.844451 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:03.844667 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:03.844697 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.844740 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:03.844769 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.847927 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:46193
I20240503 08:08:03.847961 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:46193 every 8 connection(s)
I20240503 08:08:03.848315 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:03.850787 18310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.851547 18307 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35761
I20240503 08:08:03.851621 18307 heartbeater.cc:464] Registering TS with master...
W20240503 08:08:03.851709 18311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.851755 18307 heartbeater.cc:511] Master 127.16.212.254:35761 requested a full tablet report, sending...
I20240503 08:08:03.852066 18102 ts_manager.cc:194] Registered new tserver with Master: 3493dff7b1504ed290781c4f9cea9f74 (127.16.212.194:46193)
W20240503 08:08:03.852478 18313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:03.852535 18102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59880
I20240503 08:08:03.853191 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:03.853320 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:03.853348 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:03.853359 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723683853360 us; error 0 us; skew 500 ppm
I20240503 08:08:03.854084 17235 webserver.cc:460] Webserver started at http://127.16.212.195:41259/ using document root <none> and password file <none>
I20240503 08:08:03.854171 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:03.854252 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:03.854286 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:03.854574 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "a534509e0e3048d28c11dce2d50edd41"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.855510 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:03.855970 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.856091 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.856141 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "a534509e0e3048d28c11dce2d50edd41"
format_stamp: "Formatted at 2024-05-03 08:08:03 on dist-test-slave-zl2t"
I20240503 08:08:03.856184 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:03.867455 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:03.867678 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:03.867842 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:03.868033 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:03.868064 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.868085 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:03.868113 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.871317 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:40049
I20240503 08:08:03.871394 18381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:40049 every 8 connection(s)
I20240503 08:08:03.874341 18382 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35761
I20240503 08:08:03.874398 18382 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:03.874496 18382 heartbeater.cc:511] Master 127.16.212.254:35761 requested a full tablet report, sending...
I20240503 08:08:03.874763 18102 ts_manager.cc:194] Registered new tserver with Master: a534509e0e3048d28c11dce2d50edd41 (127.16.212.195:40049)
I20240503 08:08:03.874821 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003317916s
I20240503 08:08:03.875227 18102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59886
I20240503 08:08:03.875238 17235 test_util.cc:268] Using random seed: 940620197
I20240503 08:08:03.879999 18102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59888:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:08:03.880134 18102 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:08:03.884768 18271 tablet_service.cc:1467] Processing CreateTablet for tablet 9b200e9dae2f41ef8c9da5a7765f0b95 (DEFAULT_TABLE table=test-workload [id=16625404060440288dbcdb61409dd9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:03.884915 18271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b200e9dae2f41ef8c9da5a7765f0b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:03.884949 18346 tablet_service.cc:1467] Processing CreateTablet for tablet 9b200e9dae2f41ef8c9da5a7765f0b95 (DEFAULT_TABLE table=test-workload [id=16625404060440288dbcdb61409dd9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:03.884965 18195 tablet_service.cc:1467] Processing CreateTablet for tablet 9b200e9dae2f41ef8c9da5a7765f0b95 (DEFAULT_TABLE table=test-workload [id=16625404060440288dbcdb61409dd9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:03.885025 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b200e9dae2f41ef8c9da5a7765f0b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:03.885075 18195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b200e9dae2f41ef8c9da5a7765f0b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:03.886265 18397 tablet_bootstrap.cc:492] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: Bootstrap starting.
I20240503 08:08:03.886483 18398 tablet_bootstrap.cc:492] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41: Bootstrap starting.
I20240503 08:08:03.886664 18399 tablet_bootstrap.cc:492] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74: Bootstrap starting.
I20240503 08:08:03.886715 18397 tablet_bootstrap.cc:654] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:03.886941 18398 tablet_bootstrap.cc:654] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:03.887105 18399 tablet_bootstrap.cc:654] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:03.887358 18397 tablet_bootstrap.cc:492] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: No bootstrap required, opened a new log
I20240503 08:08:03.887440 18397 ts_tablet_manager.cc:1367] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:03.887538 18398 tablet_bootstrap.cc:492] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41: No bootstrap required, opened a new log
I20240503 08:08:03.887601 18398 ts_tablet_manager.cc:1367] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:03.887630 18399 tablet_bootstrap.cc:492] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74: No bootstrap required, opened a new log
I20240503 08:08:03.887604 18397 raft_consensus.cc:348] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:03.887706 18399 ts_tablet_manager.cc:1367] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:03.887712 18397 raft_consensus.cc:374] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:03.887758 18397 raft_consensus.cc:729] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14236800ea834aa1895d2cf102699a82, State: Initialized, Role: FOLLOWER
I20240503 08:08:03.887797 18397 consensus_queue.cc:260] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:03.887842 18399 raft_consensus.cc:348] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:03.887804 18398 raft_consensus.cc:348] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:03.887897 18399 raft_consensus.cc:374] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:03.887929 18399 raft_consensus.cc:729] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3493dff7b1504ed290781c4f9cea9f74, State: Initialized, Role: FOLLOWER
I20240503 08:08:03.887954 18397 ts_tablet_manager.cc:1398] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.887995 18231 heartbeater.cc:503] Master 127.16.212.254:35761 was elected leader, sending a full tablet report...
I20240503 08:08:03.887995 18399 consensus_queue.cc:260] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:03.888181 18399 ts_tablet_manager.cc:1398] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:03.888226 18307 heartbeater.cc:503] Master 127.16.212.254:35761 was elected leader, sending a full tablet report...
I20240503 08:08:03.887905 18398 raft_consensus.cc:374] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:03.888286 18398 raft_consensus.cc:729] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a534509e0e3048d28c11dce2d50edd41, State: Initialized, Role: FOLLOWER
I20240503 08:08:03.888432 18398 consensus_queue.cc:260] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:03.888599 18398 ts_tablet_manager.cc:1398] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:03.888679 18382 heartbeater.cc:503] Master 127.16.212.254:35761 was elected leader, sending a full tablet report...
W20240503 08:08:04.074103 18232 tablet.cc:2369] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:08:04.082631 18403 raft_consensus.cc:482] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:04.082732 18403 raft_consensus.cc:504] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:04.082911 18403 leader_election.cc:288] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a534509e0e3048d28c11dce2d50edd41 (127.16.212.195:40049), 3493dff7b1504ed290781c4f9cea9f74 (127.16.212.194:46193)
I20240503 08:08:04.085481 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b200e9dae2f41ef8c9da5a7765f0b95" candidate_uuid: "14236800ea834aa1895d2cf102699a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a534509e0e3048d28c11dce2d50edd41" is_pre_election: true
I20240503 08:08:04.085554 18281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b200e9dae2f41ef8c9da5a7765f0b95" candidate_uuid: "14236800ea834aa1895d2cf102699a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3493dff7b1504ed290781c4f9cea9f74" is_pre_election: true
I20240503 08:08:04.085597 18356 raft_consensus.cc:2454] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14236800ea834aa1895d2cf102699a82 in term 0.
I20240503 08:08:04.085599 18281 raft_consensus.cc:2454] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14236800ea834aa1895d2cf102699a82 in term 0.
I20240503 08:08:04.085903 18170 leader_election.cc:302] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14236800ea834aa1895d2cf102699a82, a534509e0e3048d28c11dce2d50edd41; no voters:
I20240503 08:08:04.086028 18403 raft_consensus.cc:2781] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:04.086072 18403 raft_consensus.cc:482] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:04.086095 18403 raft_consensus.cc:3037] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:04.086759 18403 raft_consensus.cc:504] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:04.086885 18403 leader_election.cc:288] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [CANDIDATE]: Term 1 election: Requested vote from peers a534509e0e3048d28c11dce2d50edd41 (127.16.212.195:40049), 3493dff7b1504ed290781c4f9cea9f74 (127.16.212.194:46193)
I20240503 08:08:04.087045 18356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b200e9dae2f41ef8c9da5a7765f0b95" candidate_uuid: "14236800ea834aa1895d2cf102699a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a534509e0e3048d28c11dce2d50edd41"
I20240503 08:08:04.087098 18281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b200e9dae2f41ef8c9da5a7765f0b95" candidate_uuid: "14236800ea834aa1895d2cf102699a82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3493dff7b1504ed290781c4f9cea9f74"
I20240503 08:08:04.087106 18356 raft_consensus.cc:3037] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:04.087167 18281 raft_consensus.cc:3037] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:04.087827 18356 raft_consensus.cc:2454] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14236800ea834aa1895d2cf102699a82 in term 1.
I20240503 08:08:04.087827 18281 raft_consensus.cc:2454] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14236800ea834aa1895d2cf102699a82 in term 1.
I20240503 08:08:04.088069 18170 leader_election.cc:302] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14236800ea834aa1895d2cf102699a82, a534509e0e3048d28c11dce2d50edd41; no voters:
I20240503 08:08:04.088182 18403 raft_consensus.cc:2781] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:04.088328 18403 raft_consensus.cc:686] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 1 LEADER]: Becoming Leader. State: Replica: 14236800ea834aa1895d2cf102699a82, State: Running, Role: LEADER
I20240503 08:08:04.088379 18403 consensus_queue.cc:237] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } }
I20240503 08:08:04.088925 18102 catalog_manager.cc:5484] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14236800ea834aa1895d2cf102699a82 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "14236800ea834aa1895d2cf102699a82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14236800ea834aa1895d2cf102699a82" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 45303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 } health_report { overall_health: UNKNOWN } } }
I20240503 08:08:04.121624 18356 raft_consensus.cc:1261] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 1 FOLLOWER]: Refusing update from remote peer 14236800ea834aa1895d2cf102699a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:04.121650 18281 raft_consensus.cc:1261] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 1 FOLLOWER]: Refusing update from remote peer 14236800ea834aa1895d2cf102699a82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:04.121883 18407 consensus_queue.cc:1035] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3493dff7b1504ed290781c4f9cea9f74" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:04.121977 18403 consensus_queue.cc:1035] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a534509e0e3048d28c11dce2d50edd41" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:04.124136 18415 mvcc.cc:205] Tried to move back new op lower bound from 7023508210162409472 to 7023508210026504192. Current Snapshot: MvccSnapshot[applied={T|T < 7023508210160787456}]
I20240503 08:08:15.149953 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:08:15.153558 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:15.153900 17235 tablet_replica.cc:330] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82: stopping tablet replica
I20240503 08:08:15.153985 17235 raft_consensus.cc:2229] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:15.154064 17235 raft_consensus.cc:2258] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 14236800ea834aa1895d2cf102699a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:15.167753 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:08:15.168885 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:08:15.171535 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:15.171648 17235 tablet_replica.cc:330] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74: stopping tablet replica
I20240503 08:08:15.171703 17235 raft_consensus.cc:2229] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:15.171752 17235 raft_consensus.cc:2258] T 9b200e9dae2f41ef8c9da5a7765f0b95 P 3493dff7b1504ed290781c4f9cea9f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:15.191000 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:08:15.191807 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:08:15.194659 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:15.194788 17235 tablet_replica.cc:330] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41: stopping tablet replica
I20240503 08:08:15.194840 17235 raft_consensus.cc:2229] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:15.194892 17235 raft_consensus.cc:2258] T 9b200e9dae2f41ef8c9da5a7765f0b95 P a534509e0e3048d28c11dce2d50edd41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:15.208050 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:08:15.208842 17235 master.cc:537] Master@127.16.212.254:35761 shutting down...
I20240503 08:08:15.211226 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:15.211288 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:15.211323 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0725cbf7d12445139df44d6bff7b29b9: stopping tablet replica
I20240503 08:08:15.222888 17235 master.cc:552] Master@127.16.212.254:35761 shutdown complete.
[ OK ] TxnParticipantITest.TestWaitOnAbortCommit (11492 ms)
[ RUN ] TxnParticipantITest.TestProxyBasicCalls
I20240503 08:08:15.241690 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:35863
I20240503 08:08:15.241863 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:15.243855 18452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:15.244530 18453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:15.245488 18455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.246196 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:15.246299 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:15.246317 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:15.246330 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723695246330 us; error 0 us; skew 500 ppm
I20240503 08:08:15.247112 17235 webserver.cc:460] Webserver started at http://127.16.212.254:42689/ using document root <none> and password file <none>
I20240503 08:08:15.247188 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:15.247218 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:15.247253 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:15.247504 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "331d441ee4be4ad681af455b6608191f"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.248554 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:15.249286 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.249466 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:15.249531 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "331d441ee4be4ad681af455b6608191f"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.249593 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:15.261121 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:15.261314 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:15.263774 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:35863
I20240503 08:08:15.265512 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:15.274755 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:35863 every 8 connection(s)
I20240503 08:08:15.275570 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f: Bootstrap starting.
I20240503 08:08:15.275873 18513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:15.276397 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f: No bootstrap required, opened a new log
I20240503 08:08:15.276502 18513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331d441ee4be4ad681af455b6608191f" member_type: VOTER }
I20240503 08:08:15.276562 18513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:15.276576 18513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331d441ee4be4ad681af455b6608191f, State: Initialized, Role: FOLLOWER
I20240503 08:08:15.276628 18513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331d441ee4be4ad681af455b6608191f" member_type: VOTER }
I20240503 08:08:15.276690 18513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:15.276705 18513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:15.276731 18513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:15.277221 18513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331d441ee4be4ad681af455b6608191f" member_type: VOTER }
I20240503 08:08:15.277271 18513 leader_election.cc:302] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 331d441ee4be4ad681af455b6608191f; no voters:
I20240503 08:08:15.277386 18513 leader_election.cc:288] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:15.277426 18516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:15.277520 18513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:15.277549 18516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 1 LEADER]: Becoming Leader. State: Replica: 331d441ee4be4ad681af455b6608191f, State: Running, Role: LEADER
I20240503 08:08:15.277601 18516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331d441ee4be4ad681af455b6608191f" member_type: VOTER }
I20240503 08:08:15.277786 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331d441ee4be4ad681af455b6608191f. Latest consensus state: current_term: 1 leader_uuid: "331d441ee4be4ad681af455b6608191f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331d441ee4be4ad681af455b6608191f" member_type: VOTER } }
I20240503 08:08:15.277808 18518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "331d441ee4be4ad681af455b6608191f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331d441ee4be4ad681af455b6608191f" member_type: VOTER } }
I20240503 08:08:15.277915 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:15.277990 18518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:15.278273 18524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:15.278404 18524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:15.278797 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:15.278934 18524 catalog_manager.cc:1347] Generated new cluster ID: 5a7e96497e26440e8455e6e0251aeb48
I20240503 08:08:15.278971 18524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:15.293035 18524 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:15.293468 18524 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:15.302985 18524 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f: Generated new TSK 0
I20240503 08:08:15.303086 18524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:15.310720 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:15.313033 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:15.313589 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.314271 17235 server_base.cc:1034] running on GCE node
W20240503 08:08:15.315028 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.315177 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:15.315219 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:15.315234 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723695315235 us; error 0 us; skew 500 ppm
I20240503 08:08:15.315922 17235 webserver.cc:460] Webserver started at http://127.16.212.193:34967/ using document root <none> and password file <none>
I20240503 08:08:15.316010 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:15.316052 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:15.316089 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:15.316346 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "30aac353a63e4b59905ab9f9b8eef81d"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.317209 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:15.317643 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.317775 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.317822 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "30aac353a63e4b59905ab9f9b8eef81d"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.317859 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:15.330307 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:15.330479 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:15.330631 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:15.330837 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:15.330866 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.330889 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:15.330914 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.333530 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:37567
I20240503 08:08:15.333606 18605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:37567 every 8 connection(s)
I20240503 08:08:15.333878 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:15.335917 18610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:15.336841 18611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.337221 18606 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35863
I20240503 08:08:15.337281 18606 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:15.337412 18606 heartbeater.cc:511] Master 127.16.212.254:35863 requested a full tablet report, sending...
W20240503 08:08:15.337486 18613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.337689 18477 ts_manager.cc:194] Registered new tserver with Master: 30aac353a63e4b59905ab9f9b8eef81d (127.16.212.193:37567)
I20240503 08:08:15.338236 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:15.338303 18477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38868
I20240503 08:08:15.338374 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:15.338462 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:15.338495 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723695338494 us; error 0 us; skew 500 ppm
I20240503 08:08:15.339215 17235 webserver.cc:460] Webserver started at http://127.16.212.194:38039/ using document root <none> and password file <none>
I20240503 08:08:15.339303 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:15.339355 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:15.339401 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:15.339722 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "03ef8b28317345e7bea7c6f1c893b4d3"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.340585 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:15.341012 18618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.341145 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.341188 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "03ef8b28317345e7bea7c6f1c893b4d3"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.341233 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:15.351362 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:15.351574 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:15.351763 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:15.351958 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:15.351985 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.352018 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:15.352044 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.355098 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:33117
I20240503 08:08:15.355155 18681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:33117 every 8 connection(s)
I20240503 08:08:15.355474 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:15.357987 18685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:15.358856 18686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.358942 18682 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35863
I20240503 08:08:15.359037 18682 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:15.359124 18682 heartbeater.cc:511] Master 127.16.212.254:35863 requested a full tablet report, sending...
I20240503 08:08:15.359371 18477 ts_manager.cc:194] Registered new tserver with Master: 03ef8b28317345e7bea7c6f1c893b4d3 (127.16.212.194:33117)
W20240503 08:08:15.359675 18688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:15.359942 18477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38874
I20240503 08:08:15.360368 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:15.360533 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:15.360565 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:15.360587 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723695360587 us; error 0 us; skew 500 ppm
I20240503 08:08:15.361325 17235 webserver.cc:460] Webserver started at http://127.16.212.195:41443/ using document root <none> and password file <none>
I20240503 08:08:15.361418 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:15.361474 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:15.361526 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:15.361815 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "602087e29a6d4a8380e4ad91cbc95497"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.363000 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 08:08:15.363684 18693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.363842 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:08:15.363891 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "602087e29a6d4a8380e4ad91cbc95497"
format_stamp: "Formatted at 2024-05-03 08:08:15 on dist-test-slave-zl2t"
I20240503 08:08:15.363946 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:15.377614 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:15.377875 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:15.378063 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:15.378248 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:15.378298 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.378331 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:15.378350 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.381850 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:46415
I20240503 08:08:15.381909 18756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:46415 every 8 connection(s)
I20240503 08:08:15.385143 18757 heartbeater.cc:347] Connected to a master server at 127.16.212.254:35863
I20240503 08:08:15.385212 18757 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:15.385310 18757 heartbeater.cc:511] Master 127.16.212.254:35863 requested a full tablet report, sending...
I20240503 08:08:15.385560 18477 ts_manager.cc:194] Registered new tserver with Master: 602087e29a6d4a8380e4ad91cbc95497 (127.16.212.195:46415)
I20240503 08:08:15.385906 18477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38880
I20240503 08:08:15.386534 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004502621s
I20240503 08:08:15.386934 17235 test_util.cc:268] Using random seed: 952131894
I20240503 08:08:15.391957 18477 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38882:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:08:15.392091 18477 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:08:15.396741 18570 tablet_service.cc:1467] Processing CreateTablet for tablet 2649d8533f4f41b3a70de6b97bb3a79e (DEFAULT_TABLE table=test-workload [id=9f1c0103f7d44539bffb18d76f6a30c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:15.396857 18721 tablet_service.cc:1467] Processing CreateTablet for tablet 2649d8533f4f41b3a70de6b97bb3a79e (DEFAULT_TABLE table=test-workload [id=9f1c0103f7d44539bffb18d76f6a30c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:15.396883 18570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2649d8533f4f41b3a70de6b97bb3a79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:15.396947 18721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2649d8533f4f41b3a70de6b97bb3a79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:15.397032 18646 tablet_service.cc:1467] Processing CreateTablet for tablet 2649d8533f4f41b3a70de6b97bb3a79e (DEFAULT_TABLE table=test-workload [id=9f1c0103f7d44539bffb18d76f6a30c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:15.397114 18646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2649d8533f4f41b3a70de6b97bb3a79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:15.398470 18772 tablet_bootstrap.cc:492] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497: Bootstrap starting.
I20240503 08:08:15.398692 18773 tablet_bootstrap.cc:492] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3: Bootstrap starting.
I20240503 08:08:15.398962 18772 tablet_bootstrap.cc:654] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:15.399118 18773 tablet_bootstrap.cc:654] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:15.399489 18775 tablet_bootstrap.cc:492] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d: Bootstrap starting.
I20240503 08:08:15.399644 18772 tablet_bootstrap.cc:492] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497: No bootstrap required, opened a new log
I20240503 08:08:15.399654 18773 tablet_bootstrap.cc:492] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3: No bootstrap required, opened a new log
I20240503 08:08:15.399789 18773 ts_tablet_manager.cc:1367] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:15.399789 18772 ts_tablet_manager.cc:1367] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:15.399993 18772 raft_consensus.cc:348] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.400012 18775 tablet_bootstrap.cc:654] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:15.400095 18772 raft_consensus.cc:374] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:15.399993 18773 raft_consensus.cc:348] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.400128 18772 raft_consensus.cc:729] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602087e29a6d4a8380e4ad91cbc95497, State: Initialized, Role: FOLLOWER
I20240503 08:08:15.400137 18773 raft_consensus.cc:374] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:15.400213 18773 raft_consensus.cc:729] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ef8b28317345e7bea7c6f1c893b4d3, State: Initialized, Role: FOLLOWER
I20240503 08:08:15.400233 18772 consensus_queue.cc:260] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.400251 18773 consensus_queue.cc:260] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.400396 18772 ts_tablet_manager.cc:1398] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:15.400449 18682 heartbeater.cc:503] Master 127.16.212.254:35863 was elected leader, sending a full tablet report...
I20240503 08:08:15.400475 18757 heartbeater.cc:503] Master 127.16.212.254:35863 was elected leader, sending a full tablet report...
I20240503 08:08:15.400398 18773 ts_tablet_manager.cc:1398] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:15.400732 18775 tablet_bootstrap.cc:492] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d: No bootstrap required, opened a new log
I20240503 08:08:15.400777 18775 ts_tablet_manager.cc:1367] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:15.400908 18775 raft_consensus.cc:348] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.400967 18775 raft_consensus.cc:374] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:15.400988 18775 raft_consensus.cc:729] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30aac353a63e4b59905ab9f9b8eef81d, State: Initialized, Role: FOLLOWER
I20240503 08:08:15.401033 18775 consensus_queue.cc:260] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.401190 18775 ts_tablet_manager.cc:1398] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:15.401254 18606 heartbeater.cc:503] Master 127.16.212.254:35863 was elected leader, sending a full tablet report...
I20240503 08:08:15.405349 18780 raft_consensus.cc:482] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:15.405416 18780 raft_consensus.cc:504] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.405571 18780 leader_election.cc:288] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03ef8b28317345e7bea7c6f1c893b4d3 (127.16.212.194:33117), 602087e29a6d4a8380e4ad91cbc95497 (127.16.212.195:46415)
I20240503 08:08:15.408706 18731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2649d8533f4f41b3a70de6b97bb3a79e" candidate_uuid: "30aac353a63e4b59905ab9f9b8eef81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "602087e29a6d4a8380e4ad91cbc95497" is_pre_election: true
I20240503 08:08:15.408766 18656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2649d8533f4f41b3a70de6b97bb3a79e" candidate_uuid: "30aac353a63e4b59905ab9f9b8eef81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" is_pre_election: true
I20240503 08:08:15.408780 18731 raft_consensus.cc:2454] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30aac353a63e4b59905ab9f9b8eef81d in term 0.
I20240503 08:08:15.408867 18656 raft_consensus.cc:2454] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30aac353a63e4b59905ab9f9b8eef81d in term 0.
I20240503 08:08:15.408995 18546 leader_election.cc:302] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03ef8b28317345e7bea7c6f1c893b4d3, 30aac353a63e4b59905ab9f9b8eef81d; no voters:
I20240503 08:08:15.409096 18780 raft_consensus.cc:2781] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:15.409150 18780 raft_consensus.cc:482] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:15.409165 18780 raft_consensus.cc:3037] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:15.409842 18780 raft_consensus.cc:504] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.409973 18780 leader_election.cc:288] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [CANDIDATE]: Term 1 election: Requested vote from peers 03ef8b28317345e7bea7c6f1c893b4d3 (127.16.212.194:33117), 602087e29a6d4a8380e4ad91cbc95497 (127.16.212.195:46415)
I20240503 08:08:15.410128 18656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2649d8533f4f41b3a70de6b97bb3a79e" candidate_uuid: "30aac353a63e4b59905ab9f9b8eef81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03ef8b28317345e7bea7c6f1c893b4d3"
I20240503 08:08:15.410159 18731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2649d8533f4f41b3a70de6b97bb3a79e" candidate_uuid: "30aac353a63e4b59905ab9f9b8eef81d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "602087e29a6d4a8380e4ad91cbc95497"
I20240503 08:08:15.410192 18656 raft_consensus.cc:3037] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:15.410200 18731 raft_consensus.cc:3037] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:15.410815 18656 raft_consensus.cc:2454] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30aac353a63e4b59905ab9f9b8eef81d in term 1.
I20240503 08:08:15.410859 18731 raft_consensus.cc:2454] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30aac353a63e4b59905ab9f9b8eef81d in term 1.
I20240503 08:08:15.410995 18546 leader_election.cc:302] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30aac353a63e4b59905ab9f9b8eef81d, 602087e29a6d4a8380e4ad91cbc95497; no voters:
I20240503 08:08:15.411078 18780 raft_consensus.cc:2781] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:15.411185 18780 raft_consensus.cc:686] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 1 LEADER]: Becoming Leader. State: Replica: 30aac353a63e4b59905ab9f9b8eef81d, State: Running, Role: LEADER
I20240503 08:08:15.411226 18780 consensus_queue.cc:237] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } }
I20240503 08:08:15.411703 18476 catalog_manager.cc:5484] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d reported cstate change: term changed from 0 to 1, leader changed from <none> to 30aac353a63e4b59905ab9f9b8eef81d (127.16.212.193). New cstate: current_term: 1 leader_uuid: "30aac353a63e4b59905ab9f9b8eef81d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30aac353a63e4b59905ab9f9b8eef81d" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 } health_report { overall_health: UNKNOWN } } }
I20240503 08:08:15.426225 18656 raft_consensus.cc:1261] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 1 FOLLOWER]: Refusing update from remote peer 30aac353a63e4b59905ab9f9b8eef81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:15.426244 18731 raft_consensus.cc:1261] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 1 FOLLOWER]: Refusing update from remote peer 30aac353a63e4b59905ab9f9b8eef81d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:15.426486 18780 consensus_queue.cc:1035] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "602087e29a6d4a8380e4ad91cbc95497" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:15.426622 18782 consensus_queue.cc:1035] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [LEADER]: Connected to new peer: Peer: permanent_uuid: "03ef8b28317345e7bea7c6f1c893b4d3" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:25.427898 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:08:25.430742 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:25.430858 17235 tablet_replica.cc:330] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d: stopping tablet replica
I20240503 08:08:25.430913 17235 raft_consensus.cc:2229] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:25.430985 17235 raft_consensus.cc:2258] T 2649d8533f4f41b3a70de6b97bb3a79e P 30aac353a63e4b59905ab9f9b8eef81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:25.435076 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:08:25.435885 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:08:25.438458 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:25.438585 17235 tablet_replica.cc:330] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3: stopping tablet replica
I20240503 08:08:25.438690 17235 raft_consensus.cc:2229] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:25.438740 17235 raft_consensus.cc:2258] T 2649d8533f4f41b3a70de6b97bb3a79e P 03ef8b28317345e7bea7c6f1c893b4d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:25.461237 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:08:25.462280 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:08:25.464982 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:25.465112 17235 tablet_replica.cc:330] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497: stopping tablet replica
I20240503 08:08:25.465193 17235 raft_consensus.cc:2229] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:25.465238 17235 raft_consensus.cc:2258] T 2649d8533f4f41b3a70de6b97bb3a79e P 602087e29a6d4a8380e4ad91cbc95497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:25.479020 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:08:25.480000 17235 master.cc:537] Master@127.16.212.254:35863 shutting down...
I20240503 08:08:25.481869 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:25.481942 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:25.481962 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 331d441ee4be4ad681af455b6608191f: stopping tablet replica
I20240503 08:08:25.493827 17235 master.cc:552] Master@127.16.212.254:35863 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyBasicCalls (10271 ms)
[ RUN ] TxnParticipantITest.TestBeginCommitAfterFinalize
I20240503 08:08:25.513417 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:34465
I20240503 08:08:25.513584 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:25.515971 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:25.516573 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:25.517463 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.518293 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:25.518433 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:25.518476 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:25.518492 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723705518493 us; error 0 us; skew 500 ppm
I20240503 08:08:25.519351 17235 webserver.cc:460] Webserver started at http://127.16.212.254:43693/ using document root <none> and password file <none>
I20240503 08:08:25.519451 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:25.519515 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:25.519574 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:25.519884 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "1ee35f755cc24f5eb6200119b4a60eee"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.520859 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:25.521389 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.521531 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:08:25.521581 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "1ee35f755cc24f5eb6200119b4a60eee"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.521637 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:25.533202 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:25.533411 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:25.535951 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:34465
I20240503 08:08:25.545845 18885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:34465 every 8 connection(s)
I20240503 08:08:25.545948 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:25.547569 18886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee: Bootstrap starting.
I20240503 08:08:25.547857 18886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:25.548511 18886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee: No bootstrap required, opened a new log
I20240503 08:08:25.548620 18886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee35f755cc24f5eb6200119b4a60eee" member_type: VOTER }
I20240503 08:08:25.548683 18886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:25.548697 18886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee35f755cc24f5eb6200119b4a60eee, State: Initialized, Role: FOLLOWER
I20240503 08:08:25.548748 18886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee35f755cc24f5eb6200119b4a60eee" member_type: VOTER }
I20240503 08:08:25.548786 18886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:25.548806 18886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:25.548835 18886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:25.549330 18886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee35f755cc24f5eb6200119b4a60eee" member_type: VOTER }
I20240503 08:08:25.549384 18886 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee35f755cc24f5eb6200119b4a60eee; no voters:
I20240503 08:08:25.549499 18886 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:25.549562 18889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:25.549633 18886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:25.549731 18889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 1 LEADER]: Becoming Leader. State: Replica: 1ee35f755cc24f5eb6200119b4a60eee, State: Running, Role: LEADER
I20240503 08:08:25.549783 18889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee35f755cc24f5eb6200119b4a60eee" member_type: VOTER }
I20240503 08:08:25.550031 18890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee35f755cc24f5eb6200119b4a60eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee35f755cc24f5eb6200119b4a60eee" member_type: VOTER } }
I20240503 08:08:25.550040 18891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee35f755cc24f5eb6200119b4a60eee. Latest consensus state: current_term: 1 leader_uuid: "1ee35f755cc24f5eb6200119b4a60eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee35f755cc24f5eb6200119b4a60eee" member_type: VOTER } }
I20240503 08:08:25.550186 18891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:25.550403 18897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:25.550387 18890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:25.550566 18897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:25.550980 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:25.551105 18897 catalog_manager.cc:1347] Generated new cluster ID: 039da7ccabab489683110fb851106295
I20240503 08:08:25.551149 18897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:25.562659 18897 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:25.563072 18897 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:25.570175 18897 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee: Generated new TSK 0
I20240503 08:08:25.570262 18897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:25.582926 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:25.585286 18908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:25.586107 18910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.586860 17235 server_base.cc:1034] running on GCE node
W20240503 08:08:25.587446 18907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.587589 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:25.587633 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:25.587646 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723705587647 us; error 0 us; skew 500 ppm
I20240503 08:08:25.588349 17235 webserver.cc:460] Webserver started at http://127.16.212.193:43395/ using document root <none> and password file <none>
I20240503 08:08:25.588439 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:25.588492 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:25.588550 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:25.588858 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.589813 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:08:25.590358 18915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.590508 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.590560 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.590636 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:25.606568 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:25.606874 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:25.607064 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:25.607287 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:25.607322 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.607362 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:25.607391 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.610363 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:34133
I20240503 08:08:25.610404 18978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:34133 every 8 connection(s)
I20240503 08:08:25.610764 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:25.613020 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:25.613585 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.614207 18979 heartbeater.cc:347] Connected to a master server at 127.16.212.254:34465
I20240503 08:08:25.614274 18979 heartbeater.cc:464] Registering TS with master...
W20240503 08:08:25.614421 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.614447 18979 heartbeater.cc:511] Master 127.16.212.254:34465 requested a full tablet report, sending...
I20240503 08:08:25.614730 18850 ts_manager.cc:194] Registered new tserver with Master: 8b5c01e5498e4cd2810e1a31b62b6bf6 (127.16.212.193:34133)
I20240503 08:08:25.615175 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:25.615228 18850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53398
I20240503 08:08:25.615339 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:25.615377 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:25.615397 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723705615398 us; error 0 us; skew 500 ppm
I20240503 08:08:25.616058 17235 webserver.cc:460] Webserver started at http://127.16.212.194:45387/ using document root <none> and password file <none>
I20240503 08:08:25.616129 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:25.616173 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:25.616207 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:25.616526 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "f83ffec0dcf647e8bc33c54af5a1273c"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.617384 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:25.617844 18991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.618021 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.618075 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "f83ffec0dcf647e8bc33c54af5a1273c"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.618134 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:25.633468 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:25.633684 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:25.633843 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:25.634035 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:25.634064 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.634091 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:25.634119 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.637253 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:35795
I20240503 08:08:25.637295 19054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:35795 every 8 connection(s)
I20240503 08:08:25.637619 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:25.639842 19058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.640633 19055 heartbeater.cc:347] Connected to a master server at 127.16.212.254:34465
I20240503 08:08:25.640700 19055 heartbeater.cc:464] Registering TS with master...
W20240503 08:08:25.640796 19059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.640818 19055 heartbeater.cc:511] Master 127.16.212.254:34465 requested a full tablet report, sending...
I20240503 08:08:25.641062 18850 ts_manager.cc:194] Registered new tserver with Master: f83ffec0dcf647e8bc33c54af5a1273c (127.16.212.194:35795)
W20240503 08:08:25.641355 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:25.641464 18850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53410
I20240503 08:08:25.641919 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:25.642057 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:25.642088 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:25.642112 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723705642111 us; error 0 us; skew 500 ppm
I20240503 08:08:25.642876 17235 webserver.cc:460] Webserver started at http://127.16.212.195:35615/ using document root <none> and password file <none>
I20240503 08:08:25.642966 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:25.643021 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:25.643077 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:25.643368 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "b035a2f3a8a04c41a41334a828968d0b"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.644209 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:25.644620 19066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.644753 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.644795 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "b035a2f3a8a04c41a41334a828968d0b"
format_stamp: "Formatted at 2024-05-03 08:08:25 on dist-test-slave-zl2t"
I20240503 08:08:25.644848 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:25.654778 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:25.655030 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:25.655200 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:25.655396 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:25.655426 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.655460 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:25.655488 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.658730 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:44667
I20240503 08:08:25.658774 19129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:44667 every 8 connection(s)
I20240503 08:08:25.661911 19130 heartbeater.cc:347] Connected to a master server at 127.16.212.254:34465
I20240503 08:08:25.661978 19130 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:25.662199 19130 heartbeater.cc:511] Master 127.16.212.254:34465 requested a full tablet report, sending...
I20240503 08:08:25.662499 18850 ts_manager.cc:194] Registered new tserver with Master: b035a2f3a8a04c41a41334a828968d0b (127.16.212.195:44667)
I20240503 08:08:25.662876 18850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53418
I20240503 08:08:25.663404 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004505934s
I20240503 08:08:25.663779 17235 test_util.cc:268] Using random seed: 962408738
I20240503 08:08:25.668686 18850 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53430:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:08:25.668843 18850 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:08:25.673681 19019 tablet_service.cc:1467] Processing CreateTablet for tablet e0ad0339c2ab4ff09240cd35adabde99 (DEFAULT_TABLE table=test-workload [id=0e267c135ab04b3e8e7f7b94c47af597]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:25.673763 19094 tablet_service.cc:1467] Processing CreateTablet for tablet e0ad0339c2ab4ff09240cd35adabde99 (DEFAULT_TABLE table=test-workload [id=0e267c135ab04b3e8e7f7b94c47af597]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:25.673821 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ad0339c2ab4ff09240cd35adabde99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:25.673879 19094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ad0339c2ab4ff09240cd35adabde99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:25.674726 18943 tablet_service.cc:1467] Processing CreateTablet for tablet e0ad0339c2ab4ff09240cd35adabde99 (DEFAULT_TABLE table=test-workload [id=0e267c135ab04b3e8e7f7b94c47af597]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:25.674816 18943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ad0339c2ab4ff09240cd35adabde99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:25.675128 19145 tablet_bootstrap.cc:492] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b: Bootstrap starting.
I20240503 08:08:25.675372 19146 tablet_bootstrap.cc:492] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c: Bootstrap starting.
I20240503 08:08:25.675535 19145 tablet_bootstrap.cc:654] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:25.675757 19146 tablet_bootstrap.cc:654] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:25.676214 19149 tablet_bootstrap.cc:492] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6: Bootstrap starting.
I20240503 08:08:25.676322 19145 tablet_bootstrap.cc:492] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b: No bootstrap required, opened a new log
I20240503 08:08:25.676368 19145 ts_tablet_manager.cc:1367] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:25.676453 19146 tablet_bootstrap.cc:492] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c: No bootstrap required, opened a new log
I20240503 08:08:25.676489 19145 raft_consensus.cc:348] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.676493 19146 ts_tablet_manager.cc:1367] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:25.676569 19145 raft_consensus.cc:374] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:25.676594 19145 raft_consensus.cc:729] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b035a2f3a8a04c41a41334a828968d0b, State: Initialized, Role: FOLLOWER
I20240503 08:08:25.676606 19149 tablet_bootstrap.cc:654] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:25.676667 19145 consensus_queue.cc:260] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.676698 19146 raft_consensus.cc:348] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.676753 19146 raft_consensus.cc:374] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:25.676779 19146 raft_consensus.cc:729] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83ffec0dcf647e8bc33c54af5a1273c, State: Initialized, Role: FOLLOWER
I20240503 08:08:25.676827 19145 ts_tablet_manager.cc:1398] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.676827 19146 consensus_queue.cc:260] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.676928 19130 heartbeater.cc:503] Master 127.16.212.254:34465 was elected leader, sending a full tablet report...
I20240503 08:08:25.677028 19146 ts_tablet_manager.cc:1398] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.677065 19055 heartbeater.cc:503] Master 127.16.212.254:34465 was elected leader, sending a full tablet report...
I20240503 08:08:25.677460 19149 tablet_bootstrap.cc:492] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6: No bootstrap required, opened a new log
I20240503 08:08:25.677534 19149 ts_tablet_manager.cc:1367] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:25.677701 19149 raft_consensus.cc:348] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.677771 19149 raft_consensus.cc:374] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:25.677803 19149 raft_consensus.cc:729] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5c01e5498e4cd2810e1a31b62b6bf6, State: Initialized, Role: FOLLOWER
I20240503 08:08:25.677857 19149 consensus_queue.cc:260] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.677999 19149 ts_tablet_manager.cc:1398] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:25.678035 18979 heartbeater.cc:503] Master 127.16.212.254:34465 was elected leader, sending a full tablet report...
I20240503 08:08:25.990937 19153 raft_consensus.cc:482] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:25.991068 19153 raft_consensus.cc:504] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.991312 19153 leader_election.cc:288] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b035a2f3a8a04c41a41334a828968d0b (127.16.212.195:44667), f83ffec0dcf647e8bc33c54af5a1273c (127.16.212.194:35795)
I20240503 08:08:25.994239 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ad0339c2ab4ff09240cd35adabde99" candidate_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" is_pre_election: true
I20240503 08:08:25.994304 19104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ad0339c2ab4ff09240cd35adabde99" candidate_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b035a2f3a8a04c41a41334a828968d0b" is_pre_election: true
I20240503 08:08:25.994372 19029 raft_consensus.cc:2454] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5c01e5498e4cd2810e1a31b62b6bf6 in term 0.
I20240503 08:08:25.994378 19104 raft_consensus.cc:2454] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5c01e5498e4cd2810e1a31b62b6bf6 in term 0.
I20240503 08:08:25.994576 18917 leader_election.cc:302] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5c01e5498e4cd2810e1a31b62b6bf6, f83ffec0dcf647e8bc33c54af5a1273c; no voters:
I20240503 08:08:25.994771 19153 raft_consensus.cc:2781] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:25.994813 19153 raft_consensus.cc:482] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:25.994840 19153 raft_consensus.cc:3037] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:25.995575 19153 raft_consensus.cc:504] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.995739 19153 leader_election.cc:288] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [CANDIDATE]: Term 1 election: Requested vote from peers b035a2f3a8a04c41a41334a828968d0b (127.16.212.195:44667), f83ffec0dcf647e8bc33c54af5a1273c (127.16.212.194:35795)
I20240503 08:08:25.995922 19029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ad0339c2ab4ff09240cd35adabde99" candidate_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83ffec0dcf647e8bc33c54af5a1273c"
I20240503 08:08:25.995959 19104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ad0339c2ab4ff09240cd35adabde99" candidate_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b035a2f3a8a04c41a41334a828968d0b"
I20240503 08:08:25.995995 19029 raft_consensus.cc:3037] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:25.996016 19104 raft_consensus.cc:3037] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:25.996675 19029 raft_consensus.cc:2454] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5c01e5498e4cd2810e1a31b62b6bf6 in term 1.
I20240503 08:08:25.996688 19104 raft_consensus.cc:2454] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5c01e5498e4cd2810e1a31b62b6bf6 in term 1.
I20240503 08:08:25.996882 18918 leader_election.cc:302] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5c01e5498e4cd2810e1a31b62b6bf6, b035a2f3a8a04c41a41334a828968d0b; no voters:
I20240503 08:08:25.996978 19153 raft_consensus.cc:2781] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:25.997110 19153 raft_consensus.cc:686] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 8b5c01e5498e4cd2810e1a31b62b6bf6, State: Running, Role: LEADER
I20240503 08:08:25.997153 19153 consensus_queue.cc:237] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } }
I20240503 08:08:25.997679 18850 catalog_manager.cc:5484] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b5c01e5498e4cd2810e1a31b62b6bf6 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b5c01e5498e4cd2810e1a31b62b6bf6" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 34133 } health_report { overall_health: HEALTHY } } }
I20240503 08:08:26.030745 19029 raft_consensus.cc:1261] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 1 FOLLOWER]: Refusing update from remote peer 8b5c01e5498e4cd2810e1a31b62b6bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:26.030771 19104 raft_consensus.cc:1261] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 1 FOLLOWER]: Refusing update from remote peer 8b5c01e5498e4cd2810e1a31b62b6bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:26.030965 19155 consensus_queue.cc:1035] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83ffec0dcf647e8bc33c54af5a1273c" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 35795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:26.031057 19153 consensus_queue.cc:1035] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b035a2f3a8a04c41a41334a828968d0b" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:08:36.035553 19162 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43148: Illegal state: Transaction 0 commit already in progress
I20240503 08:08:36.035851 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:08:36.038296 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:36.038408 17235 tablet_replica.cc:330] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6: stopping tablet replica
I20240503 08:08:36.038465 17235 raft_consensus.cc:2229] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:36.038533 17235 raft_consensus.cc:2258] T e0ad0339c2ab4ff09240cd35adabde99 P 8b5c01e5498e4cd2810e1a31b62b6bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:36.055626 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:08:36.056571 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:08:36.059232 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:36.059367 17235 tablet_replica.cc:330] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c: stopping tablet replica
I20240503 08:08:36.059445 17235 raft_consensus.cc:2229] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:36.059485 17235 raft_consensus.cc:2258] T e0ad0339c2ab4ff09240cd35adabde99 P f83ffec0dcf647e8bc33c54af5a1273c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:36.072968 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:08:36.073864 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:08:36.076573 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:36.076699 17235 tablet_replica.cc:330] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b: stopping tablet replica
I20240503 08:08:36.076776 17235 raft_consensus.cc:2229] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:36.076817 17235 raft_consensus.cc:2258] T e0ad0339c2ab4ff09240cd35adabde99 P b035a2f3a8a04c41a41334a828968d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:36.099761 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:08:36.100817 17235 master.cc:537] Master@127.16.212.254:34465 shutting down...
I20240503 08:08:36.103056 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:36.103127 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:36.103147 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1ee35f755cc24f5eb6200119b4a60eee: stopping tablet replica
I20240503 08:08:36.114883 17235 master.cc:552] Master@127.16.212.254:34465 shutdown complete.
[ OK ] TxnParticipantITest.TestBeginCommitAfterFinalize (10620 ms)
[ RUN ] TxnParticipantITest.TestProxyErrorWhenNotBegun
I20240503 08:08:36.133980 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:38139
I20240503 08:08:36.134147 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:08:36.136547 17235 server_base.cc:1034] running on GCE node
W20240503 08:08:36.137163 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.137923 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.138861 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:36.139035 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:36.139079 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:36.139097 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723716139097 us; error 0 us; skew 500 ppm
I20240503 08:08:36.139796 17235 webserver.cc:460] Webserver started at http://127.16.212.254:34229/ using document root <none> and password file <none>
I20240503 08:08:36.139875 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:36.139948 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:36.140004 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:36.140295 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "c4eb1929e61442ad87113d7a48c13e84"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.141346 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:36.142150 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.142447 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:36.142519 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "c4eb1929e61442ad87113d7a48c13e84"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.142589 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:36.168501 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:36.168720 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:36.171294 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:38139
I20240503 08:08:36.174984 19257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:38139 every 8 connection(s)
I20240503 08:08:36.175143 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:36.176460 19258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84: Bootstrap starting.
I20240503 08:08:36.176739 19258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:36.177244 19258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84: No bootstrap required, opened a new log
I20240503 08:08:36.177350 19258 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eb1929e61442ad87113d7a48c13e84" member_type: VOTER }
I20240503 08:08:36.177407 19258 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:36.177421 19258 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4eb1929e61442ad87113d7a48c13e84, State: Initialized, Role: FOLLOWER
I20240503 08:08:36.177471 19258 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eb1929e61442ad87113d7a48c13e84" member_type: VOTER }
I20240503 08:08:36.177510 19258 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:36.177547 19258 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:36.177577 19258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:36.178050 19258 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eb1929e61442ad87113d7a48c13e84" member_type: VOTER }
I20240503 08:08:36.178104 19258 leader_election.cc:302] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4eb1929e61442ad87113d7a48c13e84; no voters:
I20240503 08:08:36.178215 19258 leader_election.cc:288] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:36.178264 19261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:36.178361 19258 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:36.178403 19261 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 1 LEADER]: Becoming Leader. State: Replica: c4eb1929e61442ad87113d7a48c13e84, State: Running, Role: LEADER
I20240503 08:08:36.178473 19261 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eb1929e61442ad87113d7a48c13e84" member_type: VOTER }
I20240503 08:08:36.178663 19261 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4eb1929e61442ad87113d7a48c13e84. Latest consensus state: current_term: 1 leader_uuid: "c4eb1929e61442ad87113d7a48c13e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eb1929e61442ad87113d7a48c13e84" member_type: VOTER } }
I20240503 08:08:36.178751 19261 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:36.178684 19263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4eb1929e61442ad87113d7a48c13e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4eb1929e61442ad87113d7a48c13e84" member_type: VOTER } }
I20240503 08:08:36.178871 19263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:36.178988 19266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:36.179155 19266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:36.179672 19266 catalog_manager.cc:1347] Generated new cluster ID: 7f46dc7592be41bab0f902162879b9a2
I20240503 08:08:36.179704 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:36.179718 19266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:36.196847 19266 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:36.197270 19266 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:36.202479 19266 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84: Generated new TSK 0
I20240503 08:08:36.202571 19266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:36.211578 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:36.213861 19279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.214484 19280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.215306 19282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:36.215984 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:36.216112 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:36.216141 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:36.216153 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723716216154 us; error 0 us; skew 500 ppm
I20240503 08:08:36.216868 17235 webserver.cc:460] Webserver started at http://127.16.212.193:36161/ using document root <none> and password file <none>
I20240503 08:08:36.216940 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:36.216974 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:36.217025 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:36.217286 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "4d3b93489d634f6caf40026430874bf3"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.218164 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:36.218597 19287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.218757 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.218803 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "4d3b93489d634f6caf40026430874bf3"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.218847 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:36.239939 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:36.240149 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:36.240312 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:36.240515 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:36.240545 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.240567 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:36.240599 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.243468 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:43075
I20240503 08:08:36.244145 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:43075 every 8 connection(s)
I20240503 08:08:36.244189 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:36.247040 19353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.247576 19354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.248706 19356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:36.249158 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:36.249379 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:36.249488 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:36.249543 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723716249543 us; error 0 us; skew 500 ppm
I20240503 08:08:36.249722 19351 heartbeater.cc:347] Connected to a master server at 127.16.212.254:38139
I20240503 08:08:36.249787 19351 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:36.249920 19351 heartbeater.cc:511] Master 127.16.212.254:38139 requested a full tablet report, sending...
I20240503 08:08:36.250265 19222 ts_manager.cc:194] Registered new tserver with Master: 4d3b93489d634f6caf40026430874bf3 (127.16.212.193:43075)
I20240503 08:08:36.250965 19222 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54196
I20240503 08:08:36.251837 17235 webserver.cc:460] Webserver started at http://127.16.212.194:34873/ using document root <none> and password file <none>
I20240503 08:08:36.251952 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:36.252012 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:36.252064 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:36.252416 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "7c8879e74248425d9bd36989de163594"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.253365 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:36.253966 19363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.254184 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:36.254233 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "7c8879e74248425d9bd36989de163594"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.254290 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:36.264047 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:36.264245 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:36.264397 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:36.264591 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:36.264621 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.264657 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:36.264683 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.267752 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:36869
I20240503 08:08:36.267787 19426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:36869 every 8 connection(s)
I20240503 08:08:36.268118 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:36.270334 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:36.270999 19431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:36.271203 19427 heartbeater.cc:347] Connected to a master server at 127.16.212.254:38139
I20240503 08:08:36.271258 19427 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:36.271360 19427 heartbeater.cc:511] Master 127.16.212.254:38139 requested a full tablet report, sending...
I20240503 08:08:36.271595 19222 ts_manager.cc:194] Registered new tserver with Master: 7c8879e74248425d9bd36989de163594 (127.16.212.194:36869)
W20240503 08:08:36.271871 19433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:36.272013 19222 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54210
I20240503 08:08:36.272468 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:36.272634 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:36.272666 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:36.272678 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723716272679 us; error 0 us; skew 500 ppm
I20240503 08:08:36.273363 17235 webserver.cc:460] Webserver started at http://127.16.212.195:34043/ using document root <none> and password file <none>
I20240503 08:08:36.273445 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:36.273494 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:36.273555 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:36.273841 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "f9c5730f2df04637b114fb829a16d240"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.274721 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:36.275163 19438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.275312 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.275357 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "f9c5730f2df04637b114fb829a16d240"
format_stamp: "Formatted at 2024-05-03 08:08:36 on dist-test-slave-zl2t"
I20240503 08:08:36.275404 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:36.294868 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:36.295114 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:36.295284 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:36.295486 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:36.295513 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.295549 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:36.295578 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:36.299041 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:33129
I20240503 08:08:36.299091 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:33129 every 8 connection(s)
I20240503 08:08:36.302229 19502 heartbeater.cc:347] Connected to a master server at 127.16.212.254:38139
I20240503 08:08:36.302294 19502 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:36.302397 19502 heartbeater.cc:511] Master 127.16.212.254:38139 requested a full tablet report, sending...
I20240503 08:08:36.302678 19222 ts_manager.cc:194] Registered new tserver with Master: f9c5730f2df04637b114fb829a16d240 (127.16.212.195:33129)
I20240503 08:08:36.303028 19222 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54218
I20240503 08:08:36.303737 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004511013s
I20240503 08:08:36.304075 17235 test_util.cc:268] Using random seed: 973049035
I20240503 08:08:36.308398 19222 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54224:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:08:36.308517 19222 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:08:36.313336 19315 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4526417fcd4c019c1b27aab50294e5 (DEFAULT_TABLE table=test-workload [id=065d32ef093d43958540b30a4e6c9daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:36.313382 19466 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4526417fcd4c019c1b27aab50294e5 (DEFAULT_TABLE table=test-workload [id=065d32ef093d43958540b30a4e6c9daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:36.313355 19391 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4526417fcd4c019c1b27aab50294e5 (DEFAULT_TABLE table=test-workload [id=065d32ef093d43958540b30a4e6c9daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:36.313494 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4526417fcd4c019c1b27aab50294e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:36.313526 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4526417fcd4c019c1b27aab50294e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:36.313526 19466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4526417fcd4c019c1b27aab50294e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:36.314848 19517 tablet_bootstrap.cc:492] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240: Bootstrap starting.
I20240503 08:08:36.315261 19519 tablet_bootstrap.cc:492] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: Bootstrap starting.
I20240503 08:08:36.315323 19517 tablet_bootstrap.cc:654] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:36.315552 19519 tablet_bootstrap.cc:654] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:36.315891 19518 tablet_bootstrap.cc:492] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: Bootstrap starting.
I20240503 08:08:36.315892 19517 tablet_bootstrap.cc:492] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240: No bootstrap required, opened a new log
I20240503 08:08:36.316077 19517 ts_tablet_manager.cc:1367] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:36.316113 19519 tablet_bootstrap.cc:492] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: No bootstrap required, opened a new log
I20240503 08:08:36.316227 19519 ts_tablet_manager.cc:1367] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:36.316308 19517 raft_consensus.cc:348] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.316396 19517 raft_consensus.cc:374] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:36.316439 19517 raft_consensus.cc:729] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c5730f2df04637b114fb829a16d240, State: Initialized, Role: FOLLOWER
I20240503 08:08:36.316365 19518 tablet_bootstrap.cc:654] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:36.316367 19519 raft_consensus.cc:348] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.316489 19517 consensus_queue.cc:260] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.316624 19519 raft_consensus.cc:374] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:36.316651 19519 raft_consensus.cc:729] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8879e74248425d9bd36989de163594, State: Initialized, Role: FOLLOWER
I20240503 08:08:36.316735 19519 consensus_queue.cc:260] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.316768 19517 ts_tablet_manager.cc:1398] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:36.316864 19502 heartbeater.cc:503] Master 127.16.212.254:38139 was elected leader, sending a full tablet report...
I20240503 08:08:36.316910 19519 ts_tablet_manager.cc:1398] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:36.316979 19427 heartbeater.cc:503] Master 127.16.212.254:38139 was elected leader, sending a full tablet report...
I20240503 08:08:36.317428 19518 tablet_bootstrap.cc:492] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: No bootstrap required, opened a new log
I20240503 08:08:36.317516 19518 ts_tablet_manager.cc:1367] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:08:36.317701 19518 raft_consensus.cc:348] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.317768 19518 raft_consensus.cc:374] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:36.317787 19518 raft_consensus.cc:729] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3b93489d634f6caf40026430874bf3, State: Initialized, Role: FOLLOWER
I20240503 08:08:36.317844 19518 consensus_queue.cc:260] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.318027 19351 heartbeater.cc:503] Master 127.16.212.254:38139 was elected leader, sending a full tablet report...
I20240503 08:08:36.318010 19518 ts_tablet_manager.cc:1398] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20240503 08:08:36.494359 19352 tablet.cc:2369] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:08:36.672868 19524 raft_consensus.cc:482] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:36.672982 19524 raft_consensus.cc:504] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.673208 19524 leader_election.cc:288] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3b93489d634f6caf40026430874bf3 (127.16.212.193:43075), f9c5730f2df04637b114fb829a16d240 (127.16.212.195:33129)
I20240503 08:08:36.675608 19522 raft_consensus.cc:482] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:36.675686 19522 raft_consensus.cc:504] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.675757 19525 raft_consensus.cc:482] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:36.675848 19522 leader_election.cc:288] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3b93489d634f6caf40026430874bf3 (127.16.212.193:43075), 7c8879e74248425d9bd36989de163594 (127.16.212.194:36869)
I20240503 08:08:36.675829 19525 raft_consensus.cc:504] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.675922 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "7c8879e74248425d9bd36989de163594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3b93489d634f6caf40026430874bf3" is_pre_election: true
I20240503 08:08:36.675979 19325 raft_consensus.cc:2454] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8879e74248425d9bd36989de163594 in term 0.
I20240503 08:08:36.676061 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "7c8879e74248425d9bd36989de163594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c5730f2df04637b114fb829a16d240" is_pre_election: true
I20240503 08:08:36.676131 19476 raft_consensus.cc:2454] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8879e74248425d9bd36989de163594 in term 0.
I20240503 08:08:36.676152 19525 leader_election.cc:288] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9c5730f2df04637b114fb829a16d240 (127.16.212.195:33129), 7c8879e74248425d9bd36989de163594 (127.16.212.194:36869)
I20240503 08:08:36.676331 19366 leader_election.cc:302] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b93489d634f6caf40026430874bf3, 7c8879e74248425d9bd36989de163594; no voters:
I20240503 08:08:36.676725 19524 raft_consensus.cc:2781] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:36.676822 19524 raft_consensus.cc:482] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:36.676885 19524 raft_consensus.cc:3037] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:36.679384 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "f9c5730f2df04637b114fb829a16d240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8879e74248425d9bd36989de163594" is_pre_election: true
I20240503 08:08:36.679422 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "4d3b93489d634f6caf40026430874bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c5730f2df04637b114fb829a16d240" is_pre_election: true
I20240503 08:08:36.679471 19476 raft_consensus.cc:2454] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3b93489d634f6caf40026430874bf3 in term 0.
I20240503 08:08:36.679503 19524 raft_consensus.cc:504] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.679591 19288 leader_election.cc:302] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b93489d634f6caf40026430874bf3, f9c5730f2df04637b114fb829a16d240; no voters:
I20240503 08:08:36.679641 19400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "4d3b93489d634f6caf40026430874bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8879e74248425d9bd36989de163594" is_pre_election: true
I20240503 08:08:36.679808 19524 leader_election.cc:288] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3b93489d634f6caf40026430874bf3 (127.16.212.193:43075), f9c5730f2df04637b114fb829a16d240 (127.16.212.195:33129)
I20240503 08:08:36.679872 19525 raft_consensus.cc:2781] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:36.679905 19525 raft_consensus.cc:482] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:36.679929 19525 raft_consensus.cc:3037] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:36.680059 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "7c8879e74248425d9bd36989de163594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c5730f2df04637b114fb829a16d240"
I20240503 08:08:36.680111 19476 raft_consensus.cc:3037] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:36.680218 19401 raft_consensus.cc:2379] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9c5730f2df04637b114fb829a16d240 in current term 1: Already voted for candidate 7c8879e74248425d9bd36989de163594 in this term.
I20240503 08:08:36.680459 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "7c8879e74248425d9bd36989de163594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3b93489d634f6caf40026430874bf3"
I20240503 08:08:36.680681 19525 raft_consensus.cc:504] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.680764 19476 raft_consensus.cc:2454] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8879e74248425d9bd36989de163594 in term 1.
I20240503 08:08:36.680792 19325 raft_consensus.cc:2379] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c8879e74248425d9bd36989de163594 in current term 1: Already voted for candidate 4d3b93489d634f6caf40026430874bf3 in this term.
I20240503 08:08:36.680871 19525 leader_election.cc:288] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [CANDIDATE]: Term 1 election: Requested vote from peers f9c5730f2df04637b114fb829a16d240 (127.16.212.195:33129), 7c8879e74248425d9bd36989de163594 (127.16.212.194:36869)
I20240503 08:08:36.680974 19364 leader_election.cc:302] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c8879e74248425d9bd36989de163594, f9c5730f2df04637b114fb829a16d240; no voters: 4d3b93489d634f6caf40026430874bf3
I20240503 08:08:36.680979 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "4d3b93489d634f6caf40026430874bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8879e74248425d9bd36989de163594"
I20240503 08:08:36.681064 19401 raft_consensus.cc:2379] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3b93489d634f6caf40026430874bf3 in current term 1: Already voted for candidate 7c8879e74248425d9bd36989de163594 in this term.
I20240503 08:08:36.681093 19524 raft_consensus.cc:2781] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:36.681171 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "4d3b93489d634f6caf40026430874bf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c5730f2df04637b114fb829a16d240"
I20240503 08:08:36.681183 19524 raft_consensus.cc:686] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 LEADER]: Becoming Leader. State: Replica: 7c8879e74248425d9bd36989de163594, State: Running, Role: LEADER
I20240503 08:08:36.681255 19476 raft_consensus.cc:2379] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3b93489d634f6caf40026430874bf3 in current term 1: Already voted for candidate 7c8879e74248425d9bd36989de163594 in this term.
I20240503 08:08:36.681275 19524 consensus_queue.cc:237] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:36.681445 19288 leader_election.cc:302] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3b93489d634f6caf40026430874bf3; no voters: 7c8879e74248425d9bd36989de163594, f9c5730f2df04637b114fb829a16d240
I20240503 08:08:36.681456 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "f9c5730f2df04637b114fb829a16d240" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3b93489d634f6caf40026430874bf3" is_pre_election: true
I20240503 08:08:36.681528 19325 raft_consensus.cc:2379] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9c5730f2df04637b114fb829a16d240 in current term 1: Already voted for candidate 4d3b93489d634f6caf40026430874bf3 in this term.
I20240503 08:08:36.681565 19525 raft_consensus.cc:2726] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:08:36.681664 19441 leader_election.cc:302] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9c5730f2df04637b114fb829a16d240; no voters: 4d3b93489d634f6caf40026430874bf3, 7c8879e74248425d9bd36989de163594
I20240503 08:08:36.681754 19522 raft_consensus.cc:2726] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:08:36.682205 19221 catalog_manager.cc:5484] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c8879e74248425d9bd36989de163594 (127.16.212.194). New cstate: current_term: 1 leader_uuid: "7c8879e74248425d9bd36989de163594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } health_report { overall_health: HEALTHY } } }
I20240503 08:08:36.755950 19476 raft_consensus.cc:1261] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 1 FOLLOWER]: Refusing update from remote peer 7c8879e74248425d9bd36989de163594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:36.755950 19325 raft_consensus.cc:1261] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Refusing update from remote peer 7c8879e74248425d9bd36989de163594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:36.756191 19530 consensus_queue.cc:1035] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:36.756273 19524 consensus_queue.cc:1035] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:47.237679 19569 raft_consensus.cc:979] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: : Instructing follower 4d3b93489d634f6caf40026430874bf3 to start an election
I20240503 08:08:47.237763 19569 raft_consensus.cc:1067] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 LEADER]: Signalling peer 4d3b93489d634f6caf40026430874bf3 to start an election
I20240503 08:08:47.237946 19324 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5"
dest_uuid: "4d3b93489d634f6caf40026430874bf3"
from {username='slave'} at 127.0.0.1:48798
I20240503 08:08:47.238023 19324 raft_consensus.cc:482] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:08:47.238042 19324 raft_consensus.cc:3037] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:47.238763 19324 raft_consensus.cc:504] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:47.238915 19324 leader_election.cc:288] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [CANDIDATE]: Term 2 election: Requested vote from peers f9c5730f2df04637b114fb829a16d240 (127.16.212.195:33129), 7c8879e74248425d9bd36989de163594 (127.16.212.194:36869)
I20240503 08:08:47.239065 19475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "4d3b93489d634f6caf40026430874bf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 13751 } } ignore_live_leader: true dest_uuid: "f9c5730f2df04637b114fb829a16d240"
I20240503 08:08:47.239069 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4526417fcd4c019c1b27aab50294e5" candidate_uuid: "4d3b93489d634f6caf40026430874bf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 13751 } } ignore_live_leader: true dest_uuid: "7c8879e74248425d9bd36989de163594"
I20240503 08:08:47.239183 19475 raft_consensus.cc:3037] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:47.239183 19401 raft_consensus.cc:3032] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:08:47.239235 19401 raft_consensus.cc:729] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7c8879e74248425d9bd36989de163594, State: Running, Role: LEADER
I20240503 08:08:47.239290 19401 consensus_queue.cc:260] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13751, Committed index: 13751, Last appended: 1.13751, Last appended by leader: 13751, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:47.239388 19401 raft_consensus.cc:3037] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:47.239995 19475 raft_consensus.cc:2454] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3b93489d634f6caf40026430874bf3 in term 2.
I20240503 08:08:47.240110 19401 raft_consensus.cc:2454] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3b93489d634f6caf40026430874bf3 in term 2.
I20240503 08:08:47.240247 19288 leader_election.cc:302] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3b93489d634f6caf40026430874bf3, f9c5730f2df04637b114fb829a16d240; no voters:
I20240503 08:08:47.240425 19572 raft_consensus.cc:2781] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:08:47.240506 19572 raft_consensus.cc:686] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3b93489d634f6caf40026430874bf3, State: Running, Role: LEADER
I20240503 08:08:47.240554 19572 consensus_queue.cc:237] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13751, Committed index: 13751, Last appended: 1.13751, Last appended by leader: 13751, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } }
I20240503 08:08:47.241168 19220 catalog_manager.cc:5484] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 reported cstate change: term changed from 1 to 2, leader changed from 7c8879e74248425d9bd36989de163594 (127.16.212.194) to 4d3b93489d634f6caf40026430874bf3 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "4d3b93489d634f6caf40026430874bf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3b93489d634f6caf40026430874bf3" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 } health_report { overall_health: UNKNOWN } } }
W20240503 08:08:47.253391 19579 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60128: Illegal state: Transaction hasn't been successfully started
W20240503 08:08:47.254014 19579 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60128: Illegal state: Transaction metadata for transaction 1 does not exist
I20240503 08:08:47.254946 19475 raft_consensus.cc:1261] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 2 FOLLOWER]: Refusing update from remote peer 4d3b93489d634f6caf40026430874bf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13751. Preceding OpId from leader: term: 2 index: 13753. (index mismatch)
I20240503 08:08:47.254942 19401 raft_consensus.cc:1261] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 2 FOLLOWER]: Refusing update from remote peer 4d3b93489d634f6caf40026430874bf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13751. Preceding OpId from leader: term: 2 index: 13753. (index mismatch)
I20240503 08:08:47.255162 19573 consensus_queue.cc:1035] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c5730f2df04637b114fb829a16d240" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13752, Last known committed idx: 13751, Time since last communication: 0.000s
I20240503 08:08:47.255244 19572 consensus_queue.cc:1035] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c8879e74248425d9bd36989de163594" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13752, Last known committed idx: 13751, Time since last communication: 0.000s
I20240503 08:08:47.256826 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:08:47.259234 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:47.259337 17235 tablet_replica.cc:330] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3: stopping tablet replica
I20240503 08:08:47.259384 17235 raft_consensus.cc:2229] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:08:47.259447 17235 raft_consensus.cc:2258] T 6f4526417fcd4c019c1b27aab50294e5 P 4d3b93489d634f6caf40026430874bf3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:47.272964 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:08:47.273803 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:08:47.276479 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:47.276587 17235 tablet_replica.cc:330] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594: stopping tablet replica
I20240503 08:08:47.276635 17235 raft_consensus.cc:2229] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:47.276690 17235 raft_consensus.cc:2258] T 6f4526417fcd4c019c1b27aab50294e5 P 7c8879e74248425d9bd36989de163594 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:47.295979 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:08:47.296823 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:08:47.299657 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:47.299759 17235 tablet_replica.cc:330] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240: stopping tablet replica
I20240503 08:08:47.299824 17235 raft_consensus.cc:2229] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:47.299881 17235 raft_consensus.cc:2258] T 6f4526417fcd4c019c1b27aab50294e5 P f9c5730f2df04637b114fb829a16d240 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:47.313166 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:08:47.314039 17235 master.cc:537] Master@127.16.212.254:38139 shutting down...
I20240503 08:08:47.316418 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:47.316494 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:47.316524 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P c4eb1929e61442ad87113d7a48c13e84: stopping tablet replica
I20240503 08:08:47.328294 17235 master.cc:552] Master@127.16.212.254:38139 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyErrorWhenNotBegun (11212 ms)
[ RUN ] TxnParticipantITest.TestProxyIllegalStatesInCommitSequence
I20240503 08:08:47.347000 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:33429
I20240503 08:08:47.347151 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.350003 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.350958 17235 server_base.cc:1034] running on GCE node
W20240503 08:08:47.351706 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.352927 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.353130 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.353174 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.353188 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723727353188 us; error 0 us; skew 500 ppm
I20240503 08:08:47.353979 17235 webserver.cc:460] Webserver started at http://127.16.212.254:39725/ using document root <none> and password file <none>
I20240503 08:08:47.354061 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.354122 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.354176 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.354465 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "8729d3ed8216408e8a0e7d1838547e51"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.355409 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:47.355916 19595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.356077 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.356124 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "8729d3ed8216408e8a0e7d1838547e51"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.356176 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.371402 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.371596 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.374101 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:33429
I20240503 08:08:47.375424 19648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:47.385895 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:33429 every 8 connection(s)
I20240503 08:08:47.386638 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51: Bootstrap starting.
I20240503 08:08:47.386919 19648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:47.387423 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51: No bootstrap required, opened a new log
I20240503 08:08:47.387526 19648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8729d3ed8216408e8a0e7d1838547e51" member_type: VOTER }
I20240503 08:08:47.387576 19648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:47.387589 19648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8729d3ed8216408e8a0e7d1838547e51, State: Initialized, Role: FOLLOWER
I20240503 08:08:47.387629 19648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8729d3ed8216408e8a0e7d1838547e51" member_type: VOTER }
I20240503 08:08:47.387672 19648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:47.387693 19648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:47.387712 19648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:47.388182 19648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8729d3ed8216408e8a0e7d1838547e51" member_type: VOTER }
I20240503 08:08:47.388233 19648 leader_election.cc:302] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8729d3ed8216408e8a0e7d1838547e51; no voters:
I20240503 08:08:47.388317 19648 leader_election.cc:288] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:47.388392 19651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:47.388455 19648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:47.388533 19651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 1 LEADER]: Becoming Leader. State: Replica: 8729d3ed8216408e8a0e7d1838547e51, State: Running, Role: LEADER
I20240503 08:08:47.388587 19651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8729d3ed8216408e8a0e7d1838547e51" member_type: VOTER }
I20240503 08:08:47.388782 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8729d3ed8216408e8a0e7d1838547e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8729d3ed8216408e8a0e7d1838547e51" member_type: VOTER } }
I20240503 08:08:47.388792 19653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8729d3ed8216408e8a0e7d1838547e51. Latest consensus state: current_term: 1 leader_uuid: "8729d3ed8216408e8a0e7d1838547e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8729d3ed8216408e8a0e7d1838547e51" member_type: VOTER } }
I20240503 08:08:47.388854 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:47.388856 19653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:47.389663 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20240503 08:08:47.389750 19667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 08:08:47.389803 19667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 08:08:47.389869 19656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:47.389994 19656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:47.390513 19656 catalog_manager.cc:1347] Generated new cluster ID: 8384d1d3d975487b8c8c16ab098abf4c
I20240503 08:08:47.390553 19656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:47.397282 19656 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:47.397684 19656 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:47.403932 19656 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51: Generated new TSK 0
I20240503 08:08:47.404009 19656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:47.405346 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.407457 19669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.408259 19670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.408892 19672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.409603 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:47.409689 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.409723 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.409736 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723727409736 us; error 0 us; skew 500 ppm
I20240503 08:08:47.410375 17235 webserver.cc:460] Webserver started at http://127.16.212.193:39137/ using document root <none> and password file <none>
I20240503 08:08:47.410454 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.410506 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.410560 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.410869 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "9fc0a54a63074a9b99c787a3db0e7960"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.411741 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:47.412212 19677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.412355 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.412400 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "9fc0a54a63074a9b99c787a3db0e7960"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.412451 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.422904 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.423108 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.423267 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.423458 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.423485 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.423521 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.423548 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.426174 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:35747
I20240503 08:08:47.426232 19740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:35747 every 8 connection(s)
I20240503 08:08:47.426538 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.428601 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:47.429396 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.429905 19741 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33429
I20240503 08:08:47.429968 19741 heartbeater.cc:464] Registering TS with master...
W20240503 08:08:47.430033 19748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.430069 19741 heartbeater.cc:511] Master 127.16.212.254:33429 requested a full tablet report, sending...
I20240503 08:08:47.430295 19612 ts_manager.cc:194] Registered new tserver with Master: 9fc0a54a63074a9b99c787a3db0e7960 (127.16.212.193:35747)
I20240503 08:08:47.430812 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:47.430933 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:08:47.430975 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44660
W20240503 08:08:47.430980 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.431052 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723727431052 us; error 0 us; skew 500 ppm
I20240503 08:08:47.431854 17235 webserver.cc:460] Webserver started at http://127.16.212.194:33235/ using document root <none> and password file <none>
I20240503 08:08:47.431927 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.431991 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.432044 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.432336 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "6c948961c8334ea38fe36ee4b75ee254"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.433176 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:47.433585 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.433717 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.433758 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "6c948961c8334ea38fe36ee4b75ee254"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.433808 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.440240 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.440434 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.440582 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.440768 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.440796 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.440829 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.440857 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.443557 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:39079
I20240503 08:08:47.443612 19816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:39079 every 8 connection(s)
I20240503 08:08:47.443919 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:47.446590 19820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.446870 19817 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33429
I20240503 08:08:47.446923 19817 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:47.447033 19817 heartbeater.cc:511] Master 127.16.212.254:33429 requested a full tablet report, sending...
I20240503 08:08:47.447400 19612 ts_manager.cc:194] Registered new tserver with Master: 6c948961c8334ea38fe36ee4b75ee254 (127.16.212.194:39079)
W20240503 08:08:47.447412 19821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.447849 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44668
W20240503 08:08:47.448290 19823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:47.449075 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:47.449208 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:47.449239 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:47.449261 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723727449261 us; error 0 us; skew 500 ppm
I20240503 08:08:47.450007 17235 webserver.cc:460] Webserver started at http://127.16.212.195:41121/ using document root <none> and password file <none>
I20240503 08:08:47.450093 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:47.450146 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:47.450201 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:47.450520 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "14f84bc043b240dd98316da8411d90d6"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.451398 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:47.451884 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.452010 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.452056 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "14f84bc043b240dd98316da8411d90d6"
format_stamp: "Formatted at 2024-05-03 08:08:47 on dist-test-slave-zl2t"
I20240503 08:08:47.452107 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:47.456818 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:47.457006 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:47.457154 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:47.457353 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:47.457387 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.457423 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:47.457458 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.460633 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:34781
I20240503 08:08:47.460713 19891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:34781 every 8 connection(s)
I20240503 08:08:47.463774 19892 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33429
I20240503 08:08:47.463830 19892 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:47.463917 19892 heartbeater.cc:511] Master 127.16.212.254:33429 requested a full tablet report, sending...
I20240503 08:08:47.464147 19612 ts_manager.cc:194] Registered new tserver with Master: 14f84bc043b240dd98316da8411d90d6 (127.16.212.195:34781)
I20240503 08:08:47.464462 19612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44678
I20240503 08:08:47.465224 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004412749s
I20240503 08:08:47.465569 17235 test_util.cc:268] Using random seed: 984210528
I20240503 08:08:47.469970 19612 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44682:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:08:47.470115 19612 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:08:47.474583 19705 tablet_service.cc:1467] Processing CreateTablet for tablet a183aff42c9e4ba8a1f58adc5d0bd628 (DEFAULT_TABLE table=test-workload [id=eedc841c38174c35b0a62572708c505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:47.474772 19705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a183aff42c9e4ba8a1f58adc5d0bd628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:47.475117 19781 tablet_service.cc:1467] Processing CreateTablet for tablet a183aff42c9e4ba8a1f58adc5d0bd628 (DEFAULT_TABLE table=test-workload [id=eedc841c38174c35b0a62572708c505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:47.475229 19781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a183aff42c9e4ba8a1f58adc5d0bd628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:47.475417 19856 tablet_service.cc:1467] Processing CreateTablet for tablet a183aff42c9e4ba8a1f58adc5d0bd628 (DEFAULT_TABLE table=test-workload [id=eedc841c38174c35b0a62572708c505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:47.475528 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a183aff42c9e4ba8a1f58adc5d0bd628. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:47.476224 19907 tablet_bootstrap.cc:492] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960: Bootstrap starting.
I20240503 08:08:47.476590 19907 tablet_bootstrap.cc:654] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:47.476634 19908 tablet_bootstrap.cc:492] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: Bootstrap starting.
I20240503 08:08:47.477077 19908 tablet_bootstrap.cc:654] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:47.477420 19909 tablet_bootstrap.cc:492] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6: Bootstrap starting.
I20240503 08:08:47.477644 19908 tablet_bootstrap.cc:492] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: No bootstrap required, opened a new log
I20240503 08:08:47.477689 19908 ts_tablet_manager.cc:1367] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:47.477727 19909 tablet_bootstrap.cc:654] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:47.477829 19908 raft_consensus.cc:348] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.477890 19908 raft_consensus.cc:374] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:47.477902 19908 raft_consensus.cc:729] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c948961c8334ea38fe36ee4b75ee254, State: Initialized, Role: FOLLOWER
I20240503 08:08:47.477968 19908 consensus_queue.cc:260] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.478132 19908 ts_tablet_manager.cc:1398] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:08:47.478219 19817 heartbeater.cc:503] Master 127.16.212.254:33429 was elected leader, sending a full tablet report...
I20240503 08:08:47.478264 19907 tablet_bootstrap.cc:492] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960: No bootstrap required, opened a new log
I20240503 08:08:47.478310 19907 ts_tablet_manager.cc:1367] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:08:47.478426 19907 raft_consensus.cc:348] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.478489 19907 raft_consensus.cc:374] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:47.478508 19907 raft_consensus.cc:729] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc0a54a63074a9b99c787a3db0e7960, State: Initialized, Role: FOLLOWER
I20240503 08:08:47.478561 19907 consensus_queue.cc:260] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.478734 19907 ts_tablet_manager.cc:1398] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:47.478765 19741 heartbeater.cc:503] Master 127.16.212.254:33429 was elected leader, sending a full tablet report...
I20240503 08:08:47.478931 19909 tablet_bootstrap.cc:492] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6: No bootstrap required, opened a new log
I20240503 08:08:47.478976 19909 ts_tablet_manager.cc:1367] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:08:47.479115 19909 raft_consensus.cc:348] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.479180 19909 raft_consensus.cc:374] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:47.479205 19909 raft_consensus.cc:729] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f84bc043b240dd98316da8411d90d6, State: Initialized, Role: FOLLOWER
I20240503 08:08:47.479254 19909 consensus_queue.cc:260] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.479414 19909 ts_tablet_manager.cc:1398] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:08:47.479496 19892 heartbeater.cc:503] Master 127.16.212.254:33429 was elected leader, sending a full tablet report...
I20240503 08:08:47.675614 19913 raft_consensus.cc:482] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:47.675746 19913 raft_consensus.cc:504] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.675953 19913 leader_election.cc:288] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fc0a54a63074a9b99c787a3db0e7960 (127.16.212.193:35747), 14f84bc043b240dd98316da8411d90d6 (127.16.212.195:34781)
I20240503 08:08:47.678682 19866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628" candidate_uuid: "6c948961c8334ea38fe36ee4b75ee254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f84bc043b240dd98316da8411d90d6" is_pre_election: true
I20240503 08:08:47.678794 19866 raft_consensus.cc:2454] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c948961c8334ea38fe36ee4b75ee254 in term 0.
I20240503 08:08:47.678686 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628" candidate_uuid: "6c948961c8334ea38fe36ee4b75ee254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc0a54a63074a9b99c787a3db0e7960" is_pre_election: true
I20240503 08:08:47.678848 19715 raft_consensus.cc:2454] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c948961c8334ea38fe36ee4b75ee254 in term 0.
I20240503 08:08:47.679036 19756 leader_election.cc:302] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c948961c8334ea38fe36ee4b75ee254, 9fc0a54a63074a9b99c787a3db0e7960; no voters:
I20240503 08:08:47.679136 19913 raft_consensus.cc:2781] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:47.679186 19913 raft_consensus.cc:482] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:47.679204 19913 raft_consensus.cc:3037] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:47.679841 19913 raft_consensus.cc:504] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.679956 19913 leader_election.cc:288] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [CANDIDATE]: Term 1 election: Requested vote from peers 9fc0a54a63074a9b99c787a3db0e7960 (127.16.212.193:35747), 14f84bc043b240dd98316da8411d90d6 (127.16.212.195:34781)
I20240503 08:08:47.680124 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628" candidate_uuid: "6c948961c8334ea38fe36ee4b75ee254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc0a54a63074a9b99c787a3db0e7960"
I20240503 08:08:47.680151 19866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628" candidate_uuid: "6c948961c8334ea38fe36ee4b75ee254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f84bc043b240dd98316da8411d90d6"
I20240503 08:08:47.680194 19715 raft_consensus.cc:3037] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:47.680199 19866 raft_consensus.cc:3037] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:47.680845 19715 raft_consensus.cc:2454] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c948961c8334ea38fe36ee4b75ee254 in term 1.
I20240503 08:08:47.680896 19866 raft_consensus.cc:2454] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c948961c8334ea38fe36ee4b75ee254 in term 1.
I20240503 08:08:47.681010 19756 leader_election.cc:302] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c948961c8334ea38fe36ee4b75ee254, 9fc0a54a63074a9b99c787a3db0e7960; no voters:
I20240503 08:08:47.681106 19913 raft_consensus.cc:2781] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:47.681279 19913 raft_consensus.cc:686] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 LEADER]: Becoming Leader. State: Replica: 6c948961c8334ea38fe36ee4b75ee254, State: Running, Role: LEADER
I20240503 08:08:47.681345 19913 consensus_queue.cc:237] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:47.682011 19611 catalog_manager.cc:5484] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c948961c8334ea38fe36ee4b75ee254 (127.16.212.194). New cstate: current_term: 1 leader_uuid: "6c948961c8334ea38fe36ee4b75ee254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } health_report { overall_health: UNKNOWN } } }
I20240503 08:08:47.708876 19715 raft_consensus.cc:1261] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 1 FOLLOWER]: Refusing update from remote peer 6c948961c8334ea38fe36ee4b75ee254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:47.708868 19866 raft_consensus.cc:1261] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 1 FOLLOWER]: Refusing update from remote peer 6c948961c8334ea38fe36ee4b75ee254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:47.709160 19917 consensus_queue.cc:1035] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:47.709259 19913 consensus_queue.cc:1035] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:47.712775 19925 mvcc.cc:205] Tried to move back new op lower bound from 7023508388694929408 to 7023508388583464960. Current Snapshot: MvccSnapshot[applied={T|T < 7023508388693835776}]
I20240503 08:08:58.277583 19956 raft_consensus.cc:979] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: : Instructing follower 9fc0a54a63074a9b99c787a3db0e7960 to start an election
I20240503 08:08:58.277670 19956 raft_consensus.cc:1067] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 LEADER]: Signalling peer 9fc0a54a63074a9b99c787a3db0e7960 to start an election
I20240503 08:08:58.277887 19715 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628"
dest_uuid: "9fc0a54a63074a9b99c787a3db0e7960"
from {username='slave'} at 127.0.0.1:55322
I20240503 08:08:58.277973 19715 raft_consensus.cc:482] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:08:58.277995 19715 raft_consensus.cc:3037] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:58.278694 19715 raft_consensus.cc:504] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:58.278862 19715 leader_election.cc:288] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [CANDIDATE]: Term 2 election: Requested vote from peers 6c948961c8334ea38fe36ee4b75ee254 (127.16.212.194:39079), 14f84bc043b240dd98316da8411d90d6 (127.16.212.195:34781)
I20240503 08:08:58.281494 19791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628" candidate_uuid: "9fc0a54a63074a9b99c787a3db0e7960" candidate_term: 2 candidate_status { last_received { term: 1 index: 13527 } } ignore_live_leader: true dest_uuid: "6c948961c8334ea38fe36ee4b75ee254"
I20240503 08:08:58.281503 19866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a183aff42c9e4ba8a1f58adc5d0bd628" candidate_uuid: "9fc0a54a63074a9b99c787a3db0e7960" candidate_term: 2 candidate_status { last_received { term: 1 index: 13527 } } ignore_live_leader: true dest_uuid: "14f84bc043b240dd98316da8411d90d6"
I20240503 08:08:58.281584 19791 raft_consensus.cc:3032] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:08:58.281623 19866 raft_consensus.cc:3037] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:58.281633 19791 raft_consensus.cc:729] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c948961c8334ea38fe36ee4b75ee254, State: Running, Role: LEADER
I20240503 08:08:58.281713 19791 consensus_queue.cc:260] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13527, Committed index: 13527, Last appended: 1.13527, Last appended by leader: 13527, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:58.281800 19791 raft_consensus.cc:3037] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:08:58.282341 19866 raft_consensus.cc:2454] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fc0a54a63074a9b99c787a3db0e7960 in term 2.
I20240503 08:08:58.282451 19791 raft_consensus.cc:2454] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fc0a54a63074a9b99c787a3db0e7960 in term 2.
I20240503 08:08:58.282526 19680 leader_election.cc:302] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f84bc043b240dd98316da8411d90d6, 9fc0a54a63074a9b99c787a3db0e7960; no voters:
I20240503 08:08:58.282763 19962 raft_consensus.cc:2781] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:08:58.282882 19962 raft_consensus.cc:686] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 2 LEADER]: Becoming Leader. State: Replica: 9fc0a54a63074a9b99c787a3db0e7960, State: Running, Role: LEADER
I20240503 08:08:58.282939 19962 consensus_queue.cc:237] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13527, Committed index: 13527, Last appended: 1.13527, Last appended by leader: 13527, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } }
I20240503 08:08:58.283349 19611 catalog_manager.cc:5484] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 reported cstate change: term changed from 1 to 2, leader changed from 6c948961c8334ea38fe36ee4b75ee254 (127.16.212.194) to 9fc0a54a63074a9b99c787a3db0e7960 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "9fc0a54a63074a9b99c787a3db0e7960" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fc0a54a63074a9b99c787a3db0e7960" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 } health_report { overall_health: UNKNOWN } } }
I20240503 08:08:58.287236 19791 raft_consensus.cc:1261] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 2 FOLLOWER]: Refusing update from remote peer 9fc0a54a63074a9b99c787a3db0e7960: Log matching property violated. Preceding OpId in replica: term: 1 index: 13527. Preceding OpId from leader: term: 2 index: 13529. (index mismatch)
I20240503 08:08:58.287246 19866 raft_consensus.cc:1261] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 2 FOLLOWER]: Refusing update from remote peer 9fc0a54a63074a9b99c787a3db0e7960: Log matching property violated. Preceding OpId in replica: term: 1 index: 13527. Preceding OpId from leader: term: 2 index: 13529. (index mismatch)
I20240503 08:08:58.287485 19963 consensus_queue.cc:1035] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f84bc043b240dd98316da8411d90d6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13528, Last known committed idx: 13527, Time since last communication: 0.000s
I20240503 08:08:58.287573 19962 consensus_queue.cc:1035] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c948961c8334ea38fe36ee4b75ee254" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13528, Last known committed idx: 13527, Time since last communication: 0.000s
W20240503 08:08:58.289376 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction 0 already open
W20240503 08:08:58.289741 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Cannot finalize transaction in state: OPEN
W20240503 08:08:58.290899 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction 0 commit already in progress
W20240503 08:08:58.291201 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:08:58.292377 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction 0 commit already in progress
W20240503 08:08:58.292666 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction metadata for transaction 0 already set
W20240503 08:08:58.292923 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:08:58.293167 19970 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42878: Illegal state: Transaction metadata for transaction 0 already exists
I20240503 08:08:58.293335 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:08:58.295876 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:58.296011 17235 tablet_replica.cc:330] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960: stopping tablet replica
I20240503 08:08:58.296062 17235 raft_consensus.cc:2229] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:08:58.296118 17235 raft_consensus.cc:2258] T a183aff42c9e4ba8a1f58adc5d0bd628 P 9fc0a54a63074a9b99c787a3db0e7960 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:58.309772 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:08:58.310676 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:08:58.313915 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:58.314064 17235 tablet_replica.cc:330] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254: stopping tablet replica
I20240503 08:08:58.314118 17235 raft_consensus.cc:2229] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:58.314153 17235 raft_consensus.cc:2258] T a183aff42c9e4ba8a1f58adc5d0bd628 P 6c948961c8334ea38fe36ee4b75ee254 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:58.335458 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:08:58.336336 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:08:58.338990 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:08:58.339102 17235 tablet_replica.cc:330] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6: stopping tablet replica
I20240503 08:08:58.339161 17235 raft_consensus.cc:2229] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:08:58.339212 17235 raft_consensus.cc:2258] T a183aff42c9e4ba8a1f58adc5d0bd628 P 14f84bc043b240dd98316da8411d90d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:58.352552 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:08:58.353793 17235 master.cc:537] Master@127.16.212.254:33429 shutting down...
I20240503 08:08:58.356395 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:08:58.356477 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:08:58.356499 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8729d3ed8216408e8a0e7d1838547e51: stopping tablet replica
I20240503 08:08:58.368497 17235 master.cc:552] Master@127.16.212.254:33429 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyIllegalStatesInCommitSequence (11039 ms)
[ RUN ] TxnParticipantITest.TestProxyIllegalStatesInAbortSequence
I20240503 08:08:58.386266 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:33313
I20240503 08:08:58.386442 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:58.388849 19981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:58.389519 19978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:58.390395 19979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:58.391059 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:58.391156 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:58.391191 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:58.391203 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723738391204 us; error 0 us; skew 500 ppm
I20240503 08:08:58.391887 17235 webserver.cc:460] Webserver started at http://127.16.212.254:32985/ using document root <none> and password file <none>
I20240503 08:08:58.391971 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:58.392024 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:58.392079 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:58.392357 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.393209 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:58.393676 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.393803 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.393855 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.393893 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:58.397835 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:58.397972 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:58.400324 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:33313
I20240503 08:08:58.405782 20038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:33313 every 8 connection(s)
I20240503 08:08:58.411526 20039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:58.412573 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c: Bootstrap starting.
I20240503 08:08:58.412815 20039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:58.413336 20039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c: No bootstrap required, opened a new log
I20240503 08:08:58.413436 20039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" member_type: VOTER }
I20240503 08:08:58.413486 20039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:58.413498 20039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a8f53c8a4142d2bab0d279a6ba2f1c, State: Initialized, Role: FOLLOWER
I20240503 08:08:58.413537 20039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" member_type: VOTER }
I20240503 08:08:58.413578 20039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:08:58.413600 20039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:08:58.413625 20039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:58.414081 20039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" member_type: VOTER }
I20240503 08:08:58.414134 20039 leader_election.cc:302] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04a8f53c8a4142d2bab0d279a6ba2f1c; no voters:
I20240503 08:08:58.414222 20039 leader_election.cc:288] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:08:58.414279 20042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:58.414361 20039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:08:58.414461 20042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 1 LEADER]: Becoming Leader. State: Replica: 04a8f53c8a4142d2bab0d279a6ba2f1c, State: Running, Role: LEADER
I20240503 08:08:58.414518 20042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" member_type: VOTER }
I20240503 08:08:58.414773 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" member_type: VOTER } }
I20240503 08:08:58.414870 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:58.414790 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04a8f53c8a4142d2bab0d279a6ba2f1c. Latest consensus state: current_term: 1 leader_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a8f53c8a4142d2bab0d279a6ba2f1c" member_type: VOTER } }
I20240503 08:08:58.414963 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [sys.catalog]: This master's current role is: LEADER
I20240503 08:08:58.415187 20049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:08:58.415328 20049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:08:58.415696 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:08:58.416010 20049 catalog_manager.cc:1347] Generated new cluster ID: cd8b1d9e40d3477d9138b9c0ca5965ab
I20240503 08:08:58.416065 20049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:08:58.420706 20049 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:08:58.421108 20049 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:08:58.424242 20049 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c: Generated new TSK 0
I20240503 08:08:58.424314 20049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:08:58.431545 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:58.433897 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:58.434464 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:58.435351 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:58.436103 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:58.436225 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:58.436254 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:58.436275 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723738436273 us; error 0 us; skew 500 ppm
I20240503 08:08:58.436998 17235 webserver.cc:460] Webserver started at http://127.16.212.193:38467/ using document root <none> and password file <none>
I20240503 08:08:58.437096 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:58.437150 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:58.437203 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:58.437508 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "58770917dea64d59b71a97914cc45210"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.438485 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:08:58.439139 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.439283 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.439332 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "58770917dea64d59b71a97914cc45210"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.439388 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:58.443142 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:58.443339 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:58.443466 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:58.443702 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:58.443753 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.443790 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:58.443826 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.446853 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:35229
I20240503 08:08:58.446897 20131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:35229 every 8 connection(s)
I20240503 08:08:58.447278 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:58.449771 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:58.450685 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:08:58.451440 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:58.451529 20132 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33313
I20240503 08:08:58.451627 20132 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:58.451762 20132 heartbeater.cc:511] Master 127.16.212.254:33313 requested a full tablet report, sending...
I20240503 08:08:58.452037 20003 ts_manager.cc:194] Registered new tserver with Master: 58770917dea64d59b71a97914cc45210 (127.16.212.193:35229)
I20240503 08:08:58.452219 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:58.452461 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:58.452488 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:58.452504 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723738452505 us; error 0 us; skew 500 ppm
I20240503 08:08:58.452839 20003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46928
I20240503 08:08:58.453609 17235 webserver.cc:460] Webserver started at http://127.16.212.194:43209/ using document root <none> and password file <none>
I20240503 08:08:58.453698 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:58.453758 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:58.453800 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:58.454114 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "e00e686a6edf48c2bd1fda3f41eecb65"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.455246 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:58.455734 20144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.455878 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.455921 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "e00e686a6edf48c2bd1fda3f41eecb65"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.455971 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:58.460000 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:58.460196 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:58.460322 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:58.460497 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:58.460525 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.460556 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:58.460585 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.463485 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:42417
I20240503 08:08:58.463522 20207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:42417 every 8 connection(s)
I20240503 08:08:58.463871 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:08:58.466363 20211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:58.467177 20208 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33313
I20240503 08:08:58.467260 20208 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:58.467360 20208 heartbeater.cc:511] Master 127.16.212.254:33313 requested a full tablet report, sending...
W20240503 08:08:58.467362 20214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:58.467624 20003 ts_manager.cc:194] Registered new tserver with Master: e00e686a6edf48c2bd1fda3f41eecb65 (127.16.212.194:42417)
I20240503 08:08:58.467993 20003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46938
W20240503 08:08:58.468022 20212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:08:58.469154 17235 server_base.cc:1034] running on GCE node
I20240503 08:08:58.469316 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:08:58.469349 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:08:58.469370 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723738469370 us; error 0 us; skew 500 ppm
I20240503 08:08:58.470088 17235 webserver.cc:460] Webserver started at http://127.16.212.195:40059/ using document root <none> and password file <none>
I20240503 08:08:58.470167 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:08:58.470221 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:08:58.470265 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:08:58.470563 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "33950a853e1843b5b6583cd4882c7e82"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.471511 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:08:58.472043 20219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.472203 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:08:58.472249 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "33950a853e1843b5b6583cd4882c7e82"
format_stamp: "Formatted at 2024-05-03 08:08:58 on dist-test-slave-zl2t"
I20240503 08:08:58.472302 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:08:58.485384 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:08:58.485625 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:08:58.485770 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:08:58.485955 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:08:58.485983 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.486006 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:08:58.486032 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:08:58.489261 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:33761
I20240503 08:08:58.489363 20282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:33761 every 8 connection(s)
I20240503 08:08:58.492432 20283 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33313
I20240503 08:08:58.492491 20283 heartbeater.cc:464] Registering TS with master...
I20240503 08:08:58.492576 20283 heartbeater.cc:511] Master 127.16.212.254:33313 requested a full tablet report, sending...
I20240503 08:08:58.492841 20003 ts_manager.cc:194] Registered new tserver with Master: 33950a853e1843b5b6583cd4882c7e82 (127.16.212.195:33761)
I20240503 08:08:58.493192 20003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46946
I20240503 08:08:58.493844 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004402961s
I20240503 08:08:58.494200 17235 test_util.cc:268] Using random seed: 995239160
I20240503 08:08:58.498402 20003 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46956:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:08:58.498536 20003 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:08:58.503026 20247 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0ca6ea8cf34558a42d2054c034ef5d (DEFAULT_TABLE table=test-workload [id=e8b9212527944599bdfb057230ead6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:58.503185 20247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0ca6ea8cf34558a42d2054c034ef5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:58.503446 20096 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0ca6ea8cf34558a42d2054c034ef5d (DEFAULT_TABLE table=test-workload [id=e8b9212527944599bdfb057230ead6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:58.503558 20096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0ca6ea8cf34558a42d2054c034ef5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:58.503722 20172 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0ca6ea8cf34558a42d2054c034ef5d (DEFAULT_TABLE table=test-workload [id=e8b9212527944599bdfb057230ead6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:08:58.503822 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0ca6ea8cf34558a42d2054c034ef5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:08:58.504640 20298 tablet_bootstrap.cc:492] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82: Bootstrap starting.
I20240503 08:08:58.504990 20298 tablet_bootstrap.cc:654] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:58.505164 20300 tablet_bootstrap.cc:492] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65: Bootstrap starting.
I20240503 08:08:58.505300 20299 tablet_bootstrap.cc:492] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210: Bootstrap starting.
I20240503 08:08:58.505561 20300 tablet_bootstrap.cc:654] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:58.505573 20298 tablet_bootstrap.cc:492] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82: No bootstrap required, opened a new log
I20240503 08:08:58.505734 20298 ts_tablet_manager.cc:1367] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:58.505762 20299 tablet_bootstrap.cc:654] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210: Neither blocks nor log segments found. Creating new log.
I20240503 08:08:58.505894 20298 raft_consensus.cc:348] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.506031 20298 raft_consensus.cc:374] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:58.506079 20298 raft_consensus.cc:729] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33950a853e1843b5b6583cd4882c7e82, State: Initialized, Role: FOLLOWER
I20240503 08:08:58.506155 20298 consensus_queue.cc:260] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.506199 20300 tablet_bootstrap.cc:492] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65: No bootstrap required, opened a new log
I20240503 08:08:58.506318 20299 tablet_bootstrap.cc:492] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210: No bootstrap required, opened a new log
I20240503 08:08:58.506333 20300 ts_tablet_manager.cc:1367] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:58.506434 20299 ts_tablet_manager.cc:1367] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:58.506389 20283 heartbeater.cc:503] Master 127.16.212.254:33313 was elected leader, sending a full tablet report...
I20240503 08:08:58.506385 20298 ts_tablet_manager.cc:1398] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:58.506584 20300 raft_consensus.cc:348] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.506687 20300 raft_consensus.cc:374] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:58.506744 20300 raft_consensus.cc:729] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00e686a6edf48c2bd1fda3f41eecb65, State: Initialized, Role: FOLLOWER
I20240503 08:08:58.506793 20299 raft_consensus.cc:348] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.506799 20300 consensus_queue.cc:260] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.506914 20299 raft_consensus.cc:374] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:08:58.506939 20299 raft_consensus.cc:729] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58770917dea64d59b71a97914cc45210, State: Initialized, Role: FOLLOWER
I20240503 08:08:58.507057 20208 heartbeater.cc:503] Master 127.16.212.254:33313 was elected leader, sending a full tablet report...
I20240503 08:08:58.507089 20300 ts_tablet_manager.cc:1398] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:08:58.507184 20299 consensus_queue.cc:260] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.507359 20299 ts_tablet_manager.cc:1398] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:08:58.507382 20132 heartbeater.cc:503] Master 127.16.212.254:33313 was elected leader, sending a full tablet report...
I20240503 08:08:58.607654 20306 raft_consensus.cc:482] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:08:58.607764 20306 raft_consensus.cc:504] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.607969 20306 leader_election.cc:288] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33950a853e1843b5b6583cd4882c7e82 (127.16.212.195:33761), e00e686a6edf48c2bd1fda3f41eecb65 (127.16.212.194:42417)
I20240503 08:08:58.610772 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ca6ea8cf34558a42d2054c034ef5d" candidate_uuid: "58770917dea64d59b71a97914cc45210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33950a853e1843b5b6583cd4882c7e82" is_pre_election: true
I20240503 08:08:58.610864 20257 raft_consensus.cc:2454] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58770917dea64d59b71a97914cc45210 in term 0.
I20240503 08:08:58.610845 20182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ca6ea8cf34558a42d2054c034ef5d" candidate_uuid: "58770917dea64d59b71a97914cc45210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" is_pre_election: true
I20240503 08:08:58.610949 20182 raft_consensus.cc:2454] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58770917dea64d59b71a97914cc45210 in term 0.
I20240503 08:08:58.611073 20070 leader_election.cc:302] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33950a853e1843b5b6583cd4882c7e82, 58770917dea64d59b71a97914cc45210; no voters:
I20240503 08:08:58.611186 20306 raft_consensus.cc:2781] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:08:58.611238 20306 raft_consensus.cc:482] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:08:58.611258 20306 raft_consensus.cc:3037] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:58.611908 20306 raft_consensus.cc:504] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.612030 20306 leader_election.cc:288] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [CANDIDATE]: Term 1 election: Requested vote from peers 33950a853e1843b5b6583cd4882c7e82 (127.16.212.195:33761), e00e686a6edf48c2bd1fda3f41eecb65 (127.16.212.194:42417)
I20240503 08:08:58.612205 20257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ca6ea8cf34558a42d2054c034ef5d" candidate_uuid: "58770917dea64d59b71a97914cc45210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33950a853e1843b5b6583cd4882c7e82"
I20240503 08:08:58.612228 20182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ca6ea8cf34558a42d2054c034ef5d" candidate_uuid: "58770917dea64d59b71a97914cc45210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e00e686a6edf48c2bd1fda3f41eecb65"
I20240503 08:08:58.612283 20257 raft_consensus.cc:3037] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:58.612286 20182 raft_consensus.cc:3037] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:08:58.612936 20257 raft_consensus.cc:2454] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58770917dea64d59b71a97914cc45210 in term 1.
I20240503 08:08:58.612948 20182 raft_consensus.cc:2454] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58770917dea64d59b71a97914cc45210 in term 1.
I20240503 08:08:58.613147 20070 leader_election.cc:302] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33950a853e1843b5b6583cd4882c7e82, 58770917dea64d59b71a97914cc45210; no voters:
I20240503 08:08:58.613269 20306 raft_consensus.cc:2781] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:08:58.613413 20306 raft_consensus.cc:686] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 1 LEADER]: Becoming Leader. State: Replica: 58770917dea64d59b71a97914cc45210, State: Running, Role: LEADER
I20240503 08:08:58.613474 20306 consensus_queue.cc:237] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } }
I20240503 08:08:58.614089 20001 catalog_manager.cc:5484] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58770917dea64d59b71a97914cc45210 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "58770917dea64d59b71a97914cc45210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58770917dea64d59b71a97914cc45210" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35229 } health_report { overall_health: HEALTHY } } }
I20240503 08:08:58.628108 20182 raft_consensus.cc:1261] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 1 FOLLOWER]: Refusing update from remote peer 58770917dea64d59b71a97914cc45210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:58.628245 20257 raft_consensus.cc:1261] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 1 FOLLOWER]: Refusing update from remote peer 58770917dea64d59b71a97914cc45210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:08:58.628331 20308 consensus_queue.cc:1035] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e00e686a6edf48c2bd1fda3f41eecb65" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:08:58.628464 20306 consensus_queue.cc:1035] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33950a853e1843b5b6583cd4882c7e82" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:09:08.632089 20315 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48500: Illegal state: Transaction 0 already open
W20240503 08:09:08.632452 20315 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48500: Illegal state: Cannot finalize transaction in state: OPEN
W20240503 08:09:08.633668 20315 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48500: Illegal state: Transaction metadata for transaction 0 already set
W20240503 08:09:08.634011 20315 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48500: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:09:08.634353 20315 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48500: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:09:08.634685 20315 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48500: Illegal state: Cannot begin committing transaction in state: INITIALIZING
I20240503 08:09:08.634908 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:09:08.637786 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:08.637892 17235 tablet_replica.cc:330] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210: stopping tablet replica
I20240503 08:09:08.637959 17235 raft_consensus.cc:2229] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:08.638005 17235 raft_consensus.cc:2258] T 7e0ca6ea8cf34558a42d2054c034ef5d P 58770917dea64d59b71a97914cc45210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:08.643126 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:09:08.643925 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:09:08.646564 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:08.646713 17235 tablet_replica.cc:330] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65: stopping tablet replica
I20240503 08:09:08.646782 17235 raft_consensus.cc:2229] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:08.646839 17235 raft_consensus.cc:2258] T 7e0ca6ea8cf34558a42d2054c034ef5d P e00e686a6edf48c2bd1fda3f41eecb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:08.649889 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:09:08.650552 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:09:08.652877 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:08.652985 17235 tablet_replica.cc:330] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82: stopping tablet replica
I20240503 08:09:08.653038 17235 raft_consensus.cc:2229] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:08.653086 17235 raft_consensus.cc:2258] T 7e0ca6ea8cf34558a42d2054c034ef5d P 33950a853e1843b5b6583cd4882c7e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:08.675850 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:09:08.676772 17235 master.cc:537] Master@127.16.212.254:33313 shutting down...
I20240503 08:09:08.679533 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:08.679608 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:08.679641 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04a8f53c8a4142d2bab0d279a6ba2f1c: stopping tablet replica
I20240503 08:09:08.691305 17235 master.cc:552] Master@127.16.212.254:33313 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyIllegalStatesInAbortSequence (10323 ms)
[ RUN ] TxnParticipantITest.TestProxyNonLeader
I20240503 08:09:08.710306 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:41385
I20240503 08:09:08.710476 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:08.712715 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:08.713408 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.714262 17235 server_base.cc:1034] running on GCE node
W20240503 08:09:08.715169 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.715333 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:08.715377 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:08.715400 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723748715400 us; error 0 us; skew 500 ppm
I20240503 08:09:08.716110 17235 webserver.cc:460] Webserver started at http://127.16.212.254:38679/ using document root <none> and password file <none>
I20240503 08:09:08.716200 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:08.716257 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:08.716315 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:08.716598 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "5bb2bcda1fec4f6cae4b6a966613be20"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.717455 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:08.717903 20351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.718039 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.718082 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "5bb2bcda1fec4f6cae4b6a966613be20"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.718134 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:08.731252 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:08.731436 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:08.733880 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:41385
I20240503 08:09:08.737035 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:41385 every 8 connection(s)
I20240503 08:09:08.740479 20404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:08.741500 20404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20: Bootstrap starting.
I20240503 08:09:08.741773 20404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:08.742297 20404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20: No bootstrap required, opened a new log
I20240503 08:09:08.742405 20404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" member_type: VOTER }
I20240503 08:09:08.742465 20404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:08.742480 20404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb2bcda1fec4f6cae4b6a966613be20, State: Initialized, Role: FOLLOWER
I20240503 08:09:08.742534 20404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" member_type: VOTER }
I20240503 08:09:08.742574 20404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:08.742594 20404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:08.742642 20404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:08.743119 20404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" member_type: VOTER }
I20240503 08:09:08.743172 20404 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bb2bcda1fec4f6cae4b6a966613be20; no voters:
I20240503 08:09:08.743268 20404 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:08.743330 20407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:08.743422 20404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:08.743506 20407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 1 LEADER]: Becoming Leader. State: Replica: 5bb2bcda1fec4f6cae4b6a966613be20, State: Running, Role: LEADER
I20240503 08:09:08.743568 20407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" member_type: VOTER }
I20240503 08:09:08.743788 20409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb2bcda1fec4f6cae4b6a966613be20. Latest consensus state: current_term: 1 leader_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" member_type: VOTER } }
I20240503 08:09:08.743769 20408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb2bcda1fec4f6cae4b6a966613be20" member_type: VOTER } }
I20240503 08:09:08.743929 20408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:08.744141 20414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:08.744207 20409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:08.744300 20414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:08.744696 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:08.744899 20414 catalog_manager.cc:1347] Generated new cluster ID: 4adbe232eb2a4dfdbde48e9f1974e89e
I20240503 08:09:08.744944 20414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:08.769451 20414 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:08.769950 20414 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:08.773216 20414 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20: Generated new TSK 0
I20240503 08:09:08.773293 20414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:08.776440 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:08.778692 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:08.779273 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:08.780138 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.780798 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:08.780920 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:08.780951 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:08.780970 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723748780970 us; error 0 us; skew 500 ppm
I20240503 08:09:08.781652 17235 webserver.cc:460] Webserver started at http://127.16.212.193:38541/ using document root <none> and password file <none>
I20240503 08:09:08.781736 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:08.781786 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:08.781834 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:08.782114 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "300af0da4fcf4287983eae140a7be356"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.783051 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:08.783520 20433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.783672 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.783716 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "300af0da4fcf4287983eae140a7be356"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.783767 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:08.791208 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:08.791411 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:08.791536 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:08.791708 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:08.791735 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.791769 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:08.791796 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.794507 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:42269
I20240503 08:09:08.794631 20496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:42269 every 8 connection(s)
I20240503 08:09:08.794915 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:08.796936 20501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:08.797613 20504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.798238 20497 heartbeater.cc:347] Connected to a master server at 127.16.212.254:41385
I20240503 08:09:08.798311 20497 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:08.798427 20497 heartbeater.cc:511] Master 127.16.212.254:41385 requested a full tablet report, sending...
W20240503 08:09:08.798488 20502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.798724 20368 ts_manager.cc:194] Registered new tserver with Master: 300af0da4fcf4287983eae140a7be356 (127.16.212.193:42269)
I20240503 08:09:08.799140 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:08.799283 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:08.799315 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:08.799332 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723748799332 us; error 0 us; skew 500 ppm
I20240503 08:09:08.799466 20368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57500
I20240503 08:09:08.800104 17235 webserver.cc:460] Webserver started at http://127.16.212.194:36605/ using document root <none> and password file <none>
I20240503 08:09:08.800185 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:08.800221 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:08.800253 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:08.800495 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "61ffac9d949140ed8541c4d9138d5c15"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.801306 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:08.801707 20509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.801857 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.801905 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "61ffac9d949140ed8541c4d9138d5c15"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.801954 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:08.805085 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:08.805255 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:08.805378 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:08.805550 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:08.805577 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.805605 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:08.805634 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.808429 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:33175
I20240503 08:09:08.808465 20572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:33175 every 8 connection(s)
I20240503 08:09:08.808782 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:08.810845 20576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:08.811658 20577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.811934 20573 heartbeater.cc:347] Connected to a master server at 127.16.212.254:41385
I20240503 08:09:08.811990 20573 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:08.812085 20573 heartbeater.cc:511] Master 127.16.212.254:41385 requested a full tablet report, sending...
I20240503 08:09:08.812319 20368 ts_manager.cc:194] Registered new tserver with Master: 61ffac9d949140ed8541c4d9138d5c15 (127.16.212.194:33175)
I20240503 08:09:08.812404 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:08.812703 20368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57514
W20240503 08:09:08.813072 20579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:08.813273 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:08.813309 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:08.813323 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723748813323 us; error 0 us; skew 500 ppm
I20240503 08:09:08.813939 17235 webserver.cc:460] Webserver started at http://127.16.212.195:44273/ using document root <none> and password file <none>
I20240503 08:09:08.814006 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:08.814044 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:08.814079 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:08.814352 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "0f9b0a47451941f6a58cf1de7fbc0a29"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.815232 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:08.815636 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.815759 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.815798 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "0f9b0a47451941f6a58cf1de7fbc0a29"
format_stamp: "Formatted at 2024-05-03 08:09:08 on dist-test-slave-zl2t"
I20240503 08:09:08.815825 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:08.843250 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:08.843488 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:08.843674 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:08.843852 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:08.843876 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.843904 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:08.843922 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.847287 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:37603
I20240503 08:09:08.847373 20647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:37603 every 8 connection(s)
I20240503 08:09:08.850656 20648 heartbeater.cc:347] Connected to a master server at 127.16.212.254:41385
I20240503 08:09:08.850724 20648 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:08.850852 20648 heartbeater.cc:511] Master 127.16.212.254:41385 requested a full tablet report, sending...
I20240503 08:09:08.851117 20368 ts_manager.cc:194] Registered new tserver with Master: 0f9b0a47451941f6a58cf1de7fbc0a29 (127.16.212.195:37603)
I20240503 08:09:08.851444 20368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57528
I20240503 08:09:08.851912 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004426588s
I20240503 08:09:08.852238 17235 test_util.cc:268] Using random seed: 1005597198
I20240503 08:09:08.857235 20368 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57530:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:08.857383 20368 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:08.861909 20537 tablet_service.cc:1467] Processing CreateTablet for tablet f7d2cee268284380adebb182d1df7a7a (DEFAULT_TABLE table=test-workload [id=d59999a3b8f34f52a096cec769ee6168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:08.862083 20537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d2cee268284380adebb182d1df7a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:08.862083 20612 tablet_service.cc:1467] Processing CreateTablet for tablet f7d2cee268284380adebb182d1df7a7a (DEFAULT_TABLE table=test-workload [id=d59999a3b8f34f52a096cec769ee6168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:08.862231 20612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d2cee268284380adebb182d1df7a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:08.862596 20461 tablet_service.cc:1467] Processing CreateTablet for tablet f7d2cee268284380adebb182d1df7a7a (DEFAULT_TABLE table=test-workload [id=d59999a3b8f34f52a096cec769ee6168]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:08.862702 20461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d2cee268284380adebb182d1df7a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:08.863512 20663 tablet_bootstrap.cc:492] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15: Bootstrap starting.
I20240503 08:09:08.863787 20664 tablet_bootstrap.cc:492] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29: Bootstrap starting.
I20240503 08:09:08.863955 20663 tablet_bootstrap.cc:654] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:08.864110 20665 tablet_bootstrap.cc:492] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356: Bootstrap starting.
I20240503 08:09:08.864189 20664 tablet_bootstrap.cc:654] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:08.864521 20665 tablet_bootstrap.cc:654] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:08.864732 20664 tablet_bootstrap.cc:492] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29: No bootstrap required, opened a new log
I20240503 08:09:08.864784 20664 ts_tablet_manager.cc:1367] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:08.864936 20664 raft_consensus.cc:348] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.865016 20664 raft_consensus.cc:374] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:08.865038 20664 raft_consensus.cc:729] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9b0a47451941f6a58cf1de7fbc0a29, State: Initialized, Role: FOLLOWER
I20240503 08:09:08.865088 20664 consensus_queue.cc:260] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.865125 20665 tablet_bootstrap.cc:492] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356: No bootstrap required, opened a new log
I20240503 08:09:08.865165 20665 ts_tablet_manager.cc:1367] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:08.865238 20664 ts_tablet_manager.cc:1398] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:08.865293 20648 heartbeater.cc:503] Master 127.16.212.254:41385 was elected leader, sending a full tablet report...
I20240503 08:09:08.865276 20665 raft_consensus.cc:348] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.865330 20665 raft_consensus.cc:374] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:08.865381 20665 raft_consensus.cc:729] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300af0da4fcf4287983eae140a7be356, State: Initialized, Role: FOLLOWER
I20240503 08:09:08.865460 20665 consensus_queue.cc:260] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.865489 20663 tablet_bootstrap.cc:492] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15: No bootstrap required, opened a new log
I20240503 08:09:08.865553 20663 ts_tablet_manager.cc:1367] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:08.865634 20665 ts_tablet_manager.cc:1398] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:08.865717 20663 raft_consensus.cc:348] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.865744 20497 heartbeater.cc:503] Master 127.16.212.254:41385 was elected leader, sending a full tablet report...
I20240503 08:09:08.865792 20663 raft_consensus.cc:374] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:08.865811 20663 raft_consensus.cc:729] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ffac9d949140ed8541c4d9138d5c15, State: Initialized, Role: FOLLOWER
I20240503 08:09:08.865877 20663 consensus_queue.cc:260] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.866027 20663 ts_tablet_manager.cc:1398] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:08.866104 20573 heartbeater.cc:503] Master 127.16.212.254:41385 was elected leader, sending a full tablet report...
I20240503 08:09:08.947047 20670 raft_consensus.cc:482] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:08.947176 20670 raft_consensus.cc:504] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.947451 20670 leader_election.cc:288] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f9b0a47451941f6a58cf1de7fbc0a29 (127.16.212.195:37603), 61ffac9d949140ed8541c4d9138d5c15 (127.16.212.194:33175)
I20240503 08:09:08.948184 20669 raft_consensus.cc:482] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:08.948276 20669 raft_consensus.cc:504] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.948493 20669 leader_election.cc:288] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 300af0da4fcf4287983eae140a7be356 (127.16.212.193:42269), 61ffac9d949140ed8541c4d9138d5c15 (127.16.212.194:33175)
I20240503 08:09:08.951261 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "300af0da4fcf4287983eae140a7be356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" is_pre_election: true
I20240503 08:09:08.951355 20547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "300af0da4fcf4287983eae140a7be356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ffac9d949140ed8541c4d9138d5c15" is_pre_election: true
I20240503 08:09:08.951368 20622 raft_consensus.cc:2454] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300af0da4fcf4287983eae140a7be356 in term 0.
I20240503 08:09:08.951824 20434 leader_election.cc:302] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9b0a47451941f6a58cf1de7fbc0a29, 300af0da4fcf4287983eae140a7be356; no voters:
I20240503 08:09:08.951937 20670 raft_consensus.cc:2781] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:08.952005 20670 raft_consensus.cc:482] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:08.951539 20547 raft_consensus.cc:2454] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 300af0da4fcf4287983eae140a7be356 in term 0.
I20240503 08:09:08.952049 20670 raft_consensus.cc:3037] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:08.952114 20546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ffac9d949140ed8541c4d9138d5c15" is_pre_election: true
I20240503 08:09:08.952188 20546 raft_consensus.cc:2454] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9b0a47451941f6a58cf1de7fbc0a29 in term 0.
I20240503 08:09:08.952275 20471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300af0da4fcf4287983eae140a7be356" is_pre_election: true
I20240503 08:09:08.952369 20588 leader_election.cc:302] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9b0a47451941f6a58cf1de7fbc0a29, 61ffac9d949140ed8541c4d9138d5c15; no voters:
I20240503 08:09:08.952446 20669 raft_consensus.cc:2781] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:08.952489 20669 raft_consensus.cc:482] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:08.952520 20669 raft_consensus.cc:3037] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:08.953065 20670 raft_consensus.cc:504] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.953241 20471 raft_consensus.cc:2379] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0f9b0a47451941f6a58cf1de7fbc0a29 in current term 1: Already voted for candidate 300af0da4fcf4287983eae140a7be356 in this term.
I20240503 08:09:08.953269 20670 leader_election.cc:288] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [CANDIDATE]: Term 1 election: Requested vote from peers 0f9b0a47451941f6a58cf1de7fbc0a29 (127.16.212.195:37603), 61ffac9d949140ed8541c4d9138d5c15 (127.16.212.194:33175)
I20240503 08:09:08.953433 20546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "300af0da4fcf4287983eae140a7be356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ffac9d949140ed8541c4d9138d5c15"
I20240503 08:09:08.953514 20546 raft_consensus.cc:3037] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:08.953235 20669 raft_consensus.cc:504] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.953444 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "300af0da4fcf4287983eae140a7be356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29"
I20240503 08:09:08.953603 20622 raft_consensus.cc:2379] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 300af0da4fcf4287983eae140a7be356 in current term 1: Already voted for candidate 0f9b0a47451941f6a58cf1de7fbc0a29 in this term.
I20240503 08:09:08.953610 20669 leader_election.cc:288] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [CANDIDATE]: Term 1 election: Requested vote from peers 300af0da4fcf4287983eae140a7be356 (127.16.212.193:42269), 61ffac9d949140ed8541c4d9138d5c15 (127.16.212.194:33175)
I20240503 08:09:08.953744 20547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ffac9d949140ed8541c4d9138d5c15"
I20240503 08:09:08.953858 20471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d2cee268284380adebb182d1df7a7a" candidate_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "300af0da4fcf4287983eae140a7be356"
I20240503 08:09:08.953922 20471 raft_consensus.cc:2379] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0f9b0a47451941f6a58cf1de7fbc0a29 in current term 1: Already voted for candidate 300af0da4fcf4287983eae140a7be356 in this term.
I20240503 08:09:08.954021 20588 leader_election.cc:302] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f9b0a47451941f6a58cf1de7fbc0a29; no voters: 300af0da4fcf4287983eae140a7be356, 61ffac9d949140ed8541c4d9138d5c15
I20240503 08:09:08.954125 20669 raft_consensus.cc:2726] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:09:08.954221 20546 raft_consensus.cc:2454] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 300af0da4fcf4287983eae140a7be356 in term 1.
I20240503 08:09:08.954478 20437 leader_election.cc:302] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 300af0da4fcf4287983eae140a7be356, 61ffac9d949140ed8541c4d9138d5c15; no voters: 0f9b0a47451941f6a58cf1de7fbc0a29
I20240503 08:09:08.954591 20670 raft_consensus.cc:2781] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:08.954751 20670 raft_consensus.cc:686] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 LEADER]: Becoming Leader. State: Replica: 300af0da4fcf4287983eae140a7be356, State: Running, Role: LEADER
I20240503 08:09:08.954797 20670 consensus_queue.cc:237] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } }
I20240503 08:09:08.955364 20368 catalog_manager.cc:5484] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 reported cstate change: term changed from 0 to 1, leader changed from <none> to 300af0da4fcf4287983eae140a7be356 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "300af0da4fcf4287983eae140a7be356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "300af0da4fcf4287983eae140a7be356" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 42269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:08.982471 20622 raft_consensus.cc:1261] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 1 FOLLOWER]: Refusing update from remote peer 300af0da4fcf4287983eae140a7be356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:08.982471 20546 raft_consensus.cc:1261] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 1 FOLLOWER]: Refusing update from remote peer 300af0da4fcf4287983eae140a7be356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:08.982780 20670 consensus_queue.cc:1035] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9b0a47451941f6a58cf1de7fbc0a29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:08.982923 20675 consensus_queue.cc:1035] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ffac9d949140ed8541c4d9138d5c15" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:09:18.988072 20537 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53808: Illegal state: replica 61ffac9d949140ed8541c4d9138d5c15 is not leader of this config: current role FOLLOWER
W20240503 08:09:18.988409 20537 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53808: Illegal state: replica 61ffac9d949140ed8541c4d9138d5c15 is not leader of this config: current role FOLLOWER
W20240503 08:09:18.988709 20537 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53808: Illegal state: replica 61ffac9d949140ed8541c4d9138d5c15 is not leader of this config: current role FOLLOWER
I20240503 08:09:18.988888 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:09:18.991377 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:18.991483 17235 tablet_replica.cc:330] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356: stopping tablet replica
I20240503 08:09:18.991534 17235 raft_consensus.cc:2229] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:18.991607 17235 raft_consensus.cc:2258] T f7d2cee268284380adebb182d1df7a7a P 300af0da4fcf4287983eae140a7be356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:19.009954 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:09:19.011060 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:09:19.014000 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:19.014111 17235 tablet_replica.cc:330] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15: stopping tablet replica
I20240503 08:09:19.014189 17235 raft_consensus.cc:2229] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:19.014262 17235 raft_consensus.cc:2258] T f7d2cee268284380adebb182d1df7a7a P 61ffac9d949140ed8541c4d9138d5c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:19.028098 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:09:19.028916 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:09:19.035921 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:19.036098 17235 tablet_replica.cc:330] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29: stopping tablet replica
I20240503 08:09:19.036185 17235 raft_consensus.cc:2229] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:19.036260 17235 raft_consensus.cc:2258] T f7d2cee268284380adebb182d1df7a7a P 0f9b0a47451941f6a58cf1de7fbc0a29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:19.057076 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:09:19.058082 17235 master.cc:537] Master@127.16.212.254:41385 shutting down...
I20240503 08:09:19.061311 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:19.061390 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:19.061420 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5bb2bcda1fec4f6cae4b6a966613be20: stopping tablet replica
I20240503 08:09:19.079280 17235 master.cc:552] Master@127.16.212.254:41385 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyNonLeader (10388 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletBootstrapping
I20240503 08:09:19.098838 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:42295
I20240503 08:09:19.099005 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:19.101650 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:19.102499 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:19.103174 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:19.103919 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:19.104034 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:19.104063 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:19.104081 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723759104082 us; error 0 us; skew 500 ppm
I20240503 08:09:19.104702 17235 webserver.cc:460] Webserver started at http://127.16.212.254:36911/ using document root <none> and password file <none>
I20240503 08:09:19.104784 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:19.104826 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:19.104877 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:19.105141 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "56cb405e4e1d4faea3dc8ca29176f3db"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.105996 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:19.106426 20729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.106537 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.106581 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "56cb405e4e1d4faea3dc8ca29176f3db"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.106647 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:19.119776 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:19.119935 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:19.122277 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:42295
I20240503 08:09:19.124928 20781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:42295 every 8 connection(s)
I20240503 08:09:19.125068 20782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.126062 20782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db: Bootstrap starting.
I20240503 08:09:19.126282 20782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:19.126888 20782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db: No bootstrap required, opened a new log
I20240503 08:09:19.126989 20782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" member_type: VOTER }
I20240503 08:09:19.127038 20782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:19.127050 20782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb405e4e1d4faea3dc8ca29176f3db, State: Initialized, Role: FOLLOWER
I20240503 08:09:19.127091 20782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" member_type: VOTER }
I20240503 08:09:19.127135 20782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:19.127162 20782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:19.127182 20782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:19.127604 20782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" member_type: VOTER }
I20240503 08:09:19.127652 20782 leader_election.cc:302] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56cb405e4e1d4faea3dc8ca29176f3db; no voters:
I20240503 08:09:19.127740 20782 leader_election.cc:288] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:19.127785 20785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:19.127876 20782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:19.127893 20785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 1 LEADER]: Becoming Leader. State: Replica: 56cb405e4e1d4faea3dc8ca29176f3db, State: Running, Role: LEADER
I20240503 08:09:19.128006 20785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" member_type: VOTER }
I20240503 08:09:19.128212 20786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" member_type: VOTER } }
I20240503 08:09:19.128221 20787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56cb405e4e1d4faea3dc8ca29176f3db. Latest consensus state: current_term: 1 leader_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb405e4e1d4faea3dc8ca29176f3db" member_type: VOTER } }
I20240503 08:09:19.128369 20786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:19.128386 20787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:19.128562 20793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:19.128695 20793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:19.129174 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:19.129231 20793 catalog_manager.cc:1347] Generated new cluster ID: 71d5a15926c54864b454efb2e3aec04d
I20240503 08:09:19.129279 20793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:19.139885 20793 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:19.140333 20793 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:19.144395 20793 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db: Generated new TSK 0
I20240503 08:09:19.144495 20793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:19.144963 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:19.147917 20804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:19.148746 20806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:19.149447 17235 server_base.cc:1034] running on GCE node
W20240503 08:09:19.150305 20803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:19.150490 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:19.150545 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:19.150559 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723759150560 us; error 0 us; skew 500 ppm
I20240503 08:09:19.151261 17235 webserver.cc:460] Webserver started at http://127.16.212.193:40147/ using document root <none> and password file <none>
I20240503 08:09:19.151345 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:19.151397 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:19.151454 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:19.151757 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "18f94441315d489c90def26d48ba0147"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.152675 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:19.153194 20811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.153321 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.153363 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "18f94441315d489c90def26d48ba0147"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.153410 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:19.177774 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:19.177979 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:19.178134 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:19.178337 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:19.178367 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.178401 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:19.178431 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.181074 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:37207
I20240503 08:09:19.181756 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:09:19.182165 20874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:37207 every 8 connection(s)
W20240503 08:09:19.183981 20877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:19.184870 20878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:19.185547 20880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:19.185576 20875 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42295
I20240503 08:09:19.185631 20875 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:19.185762 20875 heartbeater.cc:511] Master 127.16.212.254:42295 requested a full tablet report, sending...
I20240503 08:09:19.185992 20746 ts_manager.cc:194] Registered new tserver with Master: 18f94441315d489c90def26d48ba0147 (127.16.212.193:37207)
I20240503 08:09:19.186277 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:19.186381 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:19.186416 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:19.186444 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723759186444 us; error 0 us; skew 500 ppm
I20240503 08:09:19.186726 20746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58060
I20240503 08:09:19.187220 17235 webserver.cc:460] Webserver started at http://127.16.212.194:33005/ using document root <none> and password file <none>
I20240503 08:09:19.187309 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:19.187348 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:19.187383 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:19.187628 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "7bde57d864d3433598e93843b76eae57"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.188478 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:19.189003 20887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.189138 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.189190 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "7bde57d864d3433598e93843b76eae57"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.189234 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:19.219568 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:19.219797 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:19.219938 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:19.220098 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:19.220120 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.220141 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:19.220153 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.222823 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:44021
I20240503 08:09:19.222923 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:44021 every 8 connection(s)
I20240503 08:09:19.223330 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:19.225414 20955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:19.226043 20954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:19.226382 20951 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42295
I20240503 08:09:19.226437 20951 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:19.226539 20951 heartbeater.cc:511] Master 127.16.212.254:42295 requested a full tablet report, sending...
I20240503 08:09:19.226780 20746 ts_manager.cc:194] Registered new tserver with Master: 7bde57d864d3433598e93843b76eae57 (127.16.212.194:44021)
W20240503 08:09:19.226897 20957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:19.227166 20746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58068
I20240503 08:09:19.227666 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:19.227789 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:19.227826 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:19.227847 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723759227847 us; error 0 us; skew 500 ppm
I20240503 08:09:19.228565 17235 webserver.cc:460] Webserver started at http://127.16.212.195:38943/ using document root <none> and password file <none>
I20240503 08:09:19.228652 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:19.228700 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:19.228750 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:19.229013 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "69208683c30a4e9fa8e9daa3a7fcd480"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.229856 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:19.230254 20962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.230371 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.230412 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "69208683c30a4e9fa8e9daa3a7fcd480"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:19.230458 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:19.247898 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:19.248145 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:19.248430 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:19.248677 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:19.248706 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.248760 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:19.248787 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.252137 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:46507
I20240503 08:09:19.252187 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:46507 every 8 connection(s)
I20240503 08:09:19.255125 21026 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42295
I20240503 08:09:19.255182 21026 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:19.255259 21026 heartbeater.cc:511] Master 127.16.212.254:42295 requested a full tablet report, sending...
I20240503 08:09:19.255491 20746 ts_manager.cc:194] Registered new tserver with Master: 69208683c30a4e9fa8e9daa3a7fcd480 (127.16.212.195:46507)
I20240503 08:09:19.255617 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003316439s
I20240503 08:09:19.255817 20746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58070
I20240503 08:09:19.256000 17235 test_util.cc:268] Using random seed: 1016000959
I20240503 08:09:19.260340 20746 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58084:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:19.260471 20746 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:19.265013 20990 tablet_service.cc:1467] Processing CreateTablet for tablet 65b5446a7338490d8315f0483e1cecff (DEFAULT_TABLE table=test-workload [id=80c49a1b4ced4b129a2bf8abed421d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:19.265161 20990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b5446a7338490d8315f0483e1cecff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.265409 20915 tablet_service.cc:1467] Processing CreateTablet for tablet 65b5446a7338490d8315f0483e1cecff (DEFAULT_TABLE table=test-workload [id=80c49a1b4ced4b129a2bf8abed421d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:19.265411 20839 tablet_service.cc:1467] Processing CreateTablet for tablet 65b5446a7338490d8315f0483e1cecff (DEFAULT_TABLE table=test-workload [id=80c49a1b4ced4b129a2bf8abed421d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:19.265514 20915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b5446a7338490d8315f0483e1cecff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.265543 20839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b5446a7338490d8315f0483e1cecff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:19.266683 21042 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: Bootstrap starting.
I20240503 08:09:19.266963 21043 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: Bootstrap starting.
I20240503 08:09:19.267072 21042 tablet_bootstrap.cc:654] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:19.267313 21043 tablet_bootstrap.cc:654] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:19.267639 21041 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Bootstrap starting.
I20240503 08:09:19.267668 21042 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: No bootstrap required, opened a new log
I20240503 08:09:19.267715 21042 ts_tablet_manager.cc:1367] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:19.267859 21043 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: No bootstrap required, opened a new log
I20240503 08:09:19.267905 21043 ts_tablet_manager.cc:1367] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:19.267874 21042 raft_consensus.cc:348] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.268014 21041 tablet_bootstrap.cc:654] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:19.268000 21042 raft_consensus.cc:374] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:19.268069 21043 raft_consensus.cc:348] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.268190 21042 raft_consensus.cc:729] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69208683c30a4e9fa8e9daa3a7fcd480, State: Initialized, Role: FOLLOWER
I20240503 08:09:19.268235 21043 raft_consensus.cc:374] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:19.268297 21043 raft_consensus.cc:729] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bde57d864d3433598e93843b76eae57, State: Initialized, Role: FOLLOWER
I20240503 08:09:19.268332 21042 consensus_queue.cc:260] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.268378 21043 consensus_queue.cc:260] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.268541 21042 ts_tablet_manager.cc:1398] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:19.268599 20951 heartbeater.cc:503] Master 127.16.212.254:42295 was elected leader, sending a full tablet report...
I20240503 08:09:19.268565 21043 ts_tablet_manager.cc:1398] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:19.268570 21026 heartbeater.cc:503] Master 127.16.212.254:42295 was elected leader, sending a full tablet report...
I20240503 08:09:19.268934 21041 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: No bootstrap required, opened a new log
I20240503 08:09:19.268980 21041 ts_tablet_manager.cc:1367] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:19.269130 21041 raft_consensus.cc:348] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.269188 21041 raft_consensus.cc:374] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:19.269212 21041 raft_consensus.cc:729] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f94441315d489c90def26d48ba0147, State: Initialized, Role: FOLLOWER
I20240503 08:09:19.269261 21041 consensus_queue.cc:260] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.269412 21041 ts_tablet_manager.cc:1398] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:19.269431 20875 heartbeater.cc:503] Master 127.16.212.254:42295 was elected leader, sending a full tablet report...
W20240503 08:09:19.432348 20876 tablet.cc:2369] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:09:19.608228 21048 raft_consensus.cc:482] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:19.608351 21048 raft_consensus.cc:504] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.608603 21048 leader_election.cc:288] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bde57d864d3433598e93843b76eae57 (127.16.212.194:44021), 18f94441315d489c90def26d48ba0147 (127.16.212.193:37207)
I20240503 08:09:19.611268 20925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff" candidate_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bde57d864d3433598e93843b76eae57" is_pre_election: true
I20240503 08:09:19.611371 20925 raft_consensus.cc:2454] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69208683c30a4e9fa8e9daa3a7fcd480 in term 0.
I20240503 08:09:19.611534 20963 leader_election.cc:302] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69208683c30a4e9fa8e9daa3a7fcd480, 7bde57d864d3433598e93843b76eae57; no voters:
I20240503 08:09:19.611631 21048 raft_consensus.cc:2781] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:19.611696 21048 raft_consensus.cc:482] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:19.611712 21048 raft_consensus.cc:3037] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:19.611784 20849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff" candidate_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18f94441315d489c90def26d48ba0147" is_pre_election: true
I20240503 08:09:19.611841 20849 raft_consensus.cc:2454] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69208683c30a4e9fa8e9daa3a7fcd480 in term 0.
I20240503 08:09:19.612416 21048 raft_consensus.cc:504] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.612529 21048 leader_election.cc:288] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [CANDIDATE]: Term 1 election: Requested vote from peers 7bde57d864d3433598e93843b76eae57 (127.16.212.194:44021), 18f94441315d489c90def26d48ba0147 (127.16.212.193:37207)
I20240503 08:09:19.612649 20925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff" candidate_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bde57d864d3433598e93843b76eae57"
I20240503 08:09:19.612689 20849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff" candidate_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18f94441315d489c90def26d48ba0147"
I20240503 08:09:19.612715 20925 raft_consensus.cc:3037] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:19.612733 20849 raft_consensus.cc:3037] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:19.613374 20925 raft_consensus.cc:2454] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69208683c30a4e9fa8e9daa3a7fcd480 in term 1.
I20240503 08:09:19.613384 20849 raft_consensus.cc:2454] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69208683c30a4e9fa8e9daa3a7fcd480 in term 1.
I20240503 08:09:19.613543 20963 leader_election.cc:302] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f94441315d489c90def26d48ba0147, 69208683c30a4e9fa8e9daa3a7fcd480; no voters:
I20240503 08:09:19.613621 21048 raft_consensus.cc:2781] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:19.613771 21048 raft_consensus.cc:686] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 LEADER]: Becoming Leader. State: Replica: 69208683c30a4e9fa8e9daa3a7fcd480, State: Running, Role: LEADER
I20240503 08:09:19.613839 21048 consensus_queue.cc:237] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:19.614408 20745 catalog_manager.cc:5484] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69208683c30a4e9fa8e9daa3a7fcd480 (127.16.212.195). New cstate: current_term: 1 leader_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:19.712607 20849 raft_consensus.cc:1261] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 1 FOLLOWER]: Refusing update from remote peer 69208683c30a4e9fa8e9daa3a7fcd480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:19.712607 20925 raft_consensus.cc:1261] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 1 FOLLOWER]: Refusing update from remote peer 69208683c30a4e9fa8e9daa3a7fcd480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:19.712886 21048 consensus_queue.cc:1035] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:19.712982 21051 consensus_queue.cc:1035] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:19.714977 21060 mvcc.cc:205] Tried to move back new op lower bound from 7023508519782666240 to 7023508519378849792. Current Snapshot: MvccSnapshot[applied={T|T < 7023508519781400576}]
I20240503 08:09:30.129249 21077 raft_consensus.cc:979] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: : Instructing follower 18f94441315d489c90def26d48ba0147 to start an election
I20240503 08:09:30.129350 21077 raft_consensus.cc:1067] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 LEADER]: Signalling peer 18f94441315d489c90def26d48ba0147 to start an election
I20240503 08:09:30.129578 20849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff"
dest_uuid: "18f94441315d489c90def26d48ba0147"
from {username='slave'} at 127.0.0.1:33698
I20240503 08:09:30.129670 20849 raft_consensus.cc:482] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:30.129704 20849 raft_consensus.cc:3037] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:30.130398 20849 raft_consensus.cc:504] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:30.130590 20849 leader_election.cc:288] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [CANDIDATE]: Term 2 election: Requested vote from peers 7bde57d864d3433598e93843b76eae57 (127.16.212.194:44021), 69208683c30a4e9fa8e9daa3a7fcd480 (127.16.212.195:46507)
I20240503 08:09:30.133352 20924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff" candidate_uuid: "18f94441315d489c90def26d48ba0147" candidate_term: 2 candidate_status { last_received { term: 1 index: 13597 } } ignore_live_leader: true dest_uuid: "7bde57d864d3433598e93843b76eae57"
I20240503 08:09:30.133348 21000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b5446a7338490d8315f0483e1cecff" candidate_uuid: "18f94441315d489c90def26d48ba0147" candidate_term: 2 candidate_status { last_received { term: 1 index: 13597 } } ignore_live_leader: true dest_uuid: "69208683c30a4e9fa8e9daa3a7fcd480"
I20240503 08:09:30.133474 20924 raft_consensus.cc:3037] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:30.133497 21000 raft_consensus.cc:3032] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:30.133523 21000 raft_consensus.cc:729] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69208683c30a4e9fa8e9daa3a7fcd480, State: Running, Role: LEADER
I20240503 08:09:30.133587 21000 consensus_queue.cc:260] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13597, Committed index: 13597, Last appended: 1.13597, Last appended by leader: 13597, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:30.133682 21000 raft_consensus.cc:3037] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:30.134364 21000 raft_consensus.cc:2454] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18f94441315d489c90def26d48ba0147 in term 2.
I20240503 08:09:30.134364 20924 raft_consensus.cc:2454] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18f94441315d489c90def26d48ba0147 in term 2.
I20240503 08:09:30.134665 20812 leader_election.cc:302] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18f94441315d489c90def26d48ba0147, 7bde57d864d3433598e93843b76eae57; no voters:
I20240503 08:09:30.134896 21095 raft_consensus.cc:2781] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:30.135036 21095 raft_consensus.cc:686] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 LEADER]: Becoming Leader. State: Replica: 18f94441315d489c90def26d48ba0147, State: Running, Role: LEADER
I20240503 08:09:30.135100 21095 consensus_queue.cc:237] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13597, Committed index: 13597, Last appended: 1.13597, Last appended by leader: 13597, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:30.135643 20746 catalog_manager.cc:5484] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 reported cstate change: term changed from 1 to 2, leader changed from 69208683c30a4e9fa8e9daa3a7fcd480 (127.16.212.195) to 18f94441315d489c90def26d48ba0147 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "18f94441315d489c90def26d48ba0147" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } health_report { overall_health: HEALTHY } } }
I20240503 08:09:30.136844 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:09:30.139698 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:30.139806 17235 tablet_replica.cc:330] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: stopping tablet replica
I20240503 08:09:30.139873 17235 raft_consensus.cc:2229] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:09:30.139931 17235 pending_rounds.cc:62] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Trying to abort 1 pending ops.
I20240503 08:09:30.139950 17235 pending_rounds.cc:68] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Aborting op as it isn't in flight: id { term: 2 index: 13598 } timestamp: 7023508562473992192 op_type: NO_OP noop_request { }
I20240503 08:09:30.140009 17235 raft_consensus.cc:2866] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:09:30.140036 17235 raft_consensus.cc:2258] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:30.143723 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:09:30.144665 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:30.147373 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:30.148177 17235 server_base.cc:1034] running on GCE node
W20240503 08:09:30.148994 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:30.149652 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:30.149791 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:30.149824 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:30.149837 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723770149837 us; error 0 us; skew 500 ppm
I20240503 08:09:30.150854 17235 webserver.cc:460] Webserver started at http://127.16.212.193:40147/ using document root <none> and password file <none>
I20240503 08:09:30.150944 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:30.150985 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:09:30.151595 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:09:30.152073 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:30.152259 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:30.152312 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "18f94441315d489c90def26d48ba0147"
format_stamp: "Formatted at 2024-05-03 08:09:19 on dist-test-slave-zl2t"
I20240503 08:09:30.152374 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:30.174546 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:30.174794 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:30.174959 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:30.175313 21116 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:09:30.176041 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:09:30.176077 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:30.176112 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:09:30.176491 17235 ts_tablet_manager.cc:590] Registered 1 tablets
W20240503 08:09:30.176517 21116 ts_tablet_manager.cc:1346] Injecting 1000ms delay in tablet bootstrapping
I20240503 08:09:30.176522 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:30.179164 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:37207
I20240503 08:09:30.186931 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:37207 every 8 connection(s)
I20240503 08:09:30.190060 21174 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42295
I20240503 08:09:30.190140 21174 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:30.190223 21174 heartbeater.cc:511] Master 127.16.212.254:42295 requested a full tablet report, sending...
I20240503 08:09:30.190534 20746 ts_manager.cc:194] Re-registered known tserver with Master: 18f94441315d489c90def26d48ba0147 (127.16.212.193:37207)
I20240503 08:09:30.190932 20746 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36502
I20240503 08:09:30.191160 17235 tablet_server.cc:178] TabletServer@127.16.212.193:37207 shutting down...
I20240503 08:09:30.193985 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:31.178572 21116 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Bootstrap starting.
W20240503 08:09:31.682531 21179 raft_consensus.cc:659] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: failed to trigger leader election: Illegal state: leader elections are disabled
I20240503 08:09:31.943706 21116 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Bootstrap replayed 1/3 log segments. Stats: ops{read=4982 overwritten=0 applied=4979 ignored=0} inserts{seen=248900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:09:32.776350 21116 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Bootstrap replayed 2/3 log segments. Stats: ops{read=9958 overwritten=0 applied=9955 ignored=0} inserts{seen=497700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:09:33.417188 21116 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Bootstrap replayed 3/3 log segments. Stats: ops{read=13598 overwritten=0 applied=13597 ignored=0} inserts{seen=679800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:09:33.417685 21116 tablet_bootstrap.cc:492] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Bootstrap complete.
I20240503 08:09:33.422421 21116 ts_tablet_manager.cc:1367] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Time spent bootstrapping tablet: real 2.244s user 1.959s sys 0.279s
I20240503 08:09:33.422629 21116 raft_consensus.cc:348] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:33.422715 21116 raft_consensus.cc:729] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18f94441315d489c90def26d48ba0147, State: Initialized, Role: FOLLOWER
I20240503 08:09:33.422778 21116 consensus_queue.cc:260] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13597, Last appended: 2.13598, Last appended by leader: 13598, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bde57d864d3433598e93843b76eae57" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 44021 } } peers { permanent_uuid: "69208683c30a4e9fa8e9daa3a7fcd480" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46507 } } peers { permanent_uuid: "18f94441315d489c90def26d48ba0147" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37207 } }
I20240503 08:09:33.422957 21116 ts_tablet_manager.cc:1398] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Time spent starting tablet: real 0.000s user 0.000s sys 0.004s
I20240503 08:09:33.423107 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:09:33.423185 17235 raft_consensus.cc:2229] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:33.423236 17235 pending_rounds.cc:62] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Trying to abort 1 pending ops.
I20240503 08:09:33.423264 17235 pending_rounds.cc:68] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147: Aborting op as it isn't in flight: id { term: 2 index: 13598 } timestamp: 7023508562473992192 op_type: NO_OP noop_request { }
I20240503 08:09:33.423313 17235 raft_consensus.cc:2866] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:09:33.423353 17235 raft_consensus.cc:2258] T 65b5446a7338490d8315f0483e1cecff P 18f94441315d489c90def26d48ba0147 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:33.438259 17235 tablet_server.cc:194] TabletServer@127.16.212.193:37207 shutdown complete.
I20240503 08:09:33.439349 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:09:33.442595 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:33.442771 17235 tablet_replica.cc:330] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57: stopping tablet replica
I20240503 08:09:33.442829 17235 raft_consensus.cc:2229] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:33.442898 17235 raft_consensus.cc:2258] T 65b5446a7338490d8315f0483e1cecff P 7bde57d864d3433598e93843b76eae57 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:33.456274 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:09:33.457273 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:09:33.460704 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:33.460832 17235 tablet_replica.cc:330] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480: stopping tablet replica
I20240503 08:09:33.460911 17235 raft_consensus.cc:2229] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:33.460947 17235 raft_consensus.cc:2258] T 65b5446a7338490d8315f0483e1cecff P 69208683c30a4e9fa8e9daa3a7fcd480 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:33.484274 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:09:33.485139 17235 master.cc:537] Master@127.16.212.254:42295 shutting down...
I20240503 08:09:33.487630 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:33.487711 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:33.487731 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56cb405e4e1d4faea3dc8ca29176f3db: stopping tablet replica
I20240503 08:09:33.499439 17235 master.cc:552] Master@127.16.212.254:42295 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletBootstrapping (14420 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletNotRunning
I20240503 08:09:33.519089 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:37363
I20240503 08:09:33.519258 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:33.521471 21185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.522145 17235 server_base.cc:1034] running on GCE node
W20240503 08:09:33.523002 21188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:33.523711 21186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.523873 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:33.523909 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:33.523922 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723773523922 us; error 0 us; skew 500 ppm
I20240503 08:09:33.524559 17235 webserver.cc:460] Webserver started at http://127.16.212.254:38025/ using document root <none> and password file <none>
I20240503 08:09:33.524647 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:33.524694 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:33.524750 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:33.525025 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "67f13766f1944a74a1c865cba4b30c19"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.525929 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:33.526396 21193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.526525 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.526569 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "67f13766f1944a74a1c865cba4b30c19"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.526634 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:33.545528 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:33.545742 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:33.548341 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:37363
I20240503 08:09:33.554586 21245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:37363 every 8 connection(s)
I20240503 08:09:33.555725 21246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:33.556797 21246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19: Bootstrap starting.
I20240503 08:09:33.557080 21246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:33.557598 21246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19: No bootstrap required, opened a new log
I20240503 08:09:33.557708 21246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f13766f1944a74a1c865cba4b30c19" member_type: VOTER }
I20240503 08:09:33.557770 21246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:33.557782 21246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f13766f1944a74a1c865cba4b30c19, State: Initialized, Role: FOLLOWER
I20240503 08:09:33.557833 21246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f13766f1944a74a1c865cba4b30c19" member_type: VOTER }
I20240503 08:09:33.557868 21246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:33.557893 21246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:33.557930 21246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:33.558429 21246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f13766f1944a74a1c865cba4b30c19" member_type: VOTER }
I20240503 08:09:33.558482 21246 leader_election.cc:302] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67f13766f1944a74a1c865cba4b30c19; no voters:
I20240503 08:09:33.558586 21246 leader_election.cc:288] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:33.558688 21249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:33.558825 21246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:33.558836 21249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 1 LEADER]: Becoming Leader. State: Replica: 67f13766f1944a74a1c865cba4b30c19, State: Running, Role: LEADER
I20240503 08:09:33.558982 21249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f13766f1944a74a1c865cba4b30c19" member_type: VOTER }
I20240503 08:09:33.559218 21250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67f13766f1944a74a1c865cba4b30c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f13766f1944a74a1c865cba4b30c19" member_type: VOTER } }
I20240503 08:09:33.559219 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67f13766f1944a74a1c865cba4b30c19. Latest consensus state: current_term: 1 leader_uuid: "67f13766f1944a74a1c865cba4b30c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f13766f1944a74a1c865cba4b30c19" member_type: VOTER } }
I20240503 08:09:33.559357 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:33.559551 21250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:33.559566 21255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:33.559746 21255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:33.560163 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:33.560335 21255 catalog_manager.cc:1347] Generated new cluster ID: 0a168de570a5497dbc3ce636e021bf0b
I20240503 08:09:33.560384 21255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:33.574995 21255 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:33.575513 21255 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:33.590312 21255 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19: Generated new TSK 0
I20240503 08:09:33.590418 21255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:33.591939 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:33.594194 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:33.595064 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:33.595752 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.596551 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:33.596649 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:33.596683 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:33.596702 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723773596702 us; error 0 us; skew 500 ppm
I20240503 08:09:33.597404 17235 webserver.cc:460] Webserver started at http://127.16.212.193:35385/ using document root <none> and password file <none>
I20240503 08:09:33.597474 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:33.597512 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:33.597550 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:33.597802 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "567ca14f62514273b683be3845f21950"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.598685 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 08:09:33.599115 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.599244 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.599303 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "567ca14f62514273b683be3845f21950"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.599339 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:33.624457 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:33.624691 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:33.624828 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:33.625010 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:33.625041 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.625062 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:33.625077 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.627995 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:41885
I20240503 08:09:33.628388 21338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:41885 every 8 connection(s)
I20240503 08:09:33.628700 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:33.631063 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:33.632045 21343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.632612 21339 heartbeater.cc:347] Connected to a master server at 127.16.212.254:37363
I20240503 08:09:33.632680 21339 heartbeater.cc:464] Registering TS with master...
W20240503 08:09:33.632722 21346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.632800 21339 heartbeater.cc:511] Master 127.16.212.254:37363 requested a full tablet report, sending...
I20240503 08:09:33.633013 21210 ts_manager.cc:194] Registered new tserver with Master: 567ca14f62514273b683be3845f21950 (127.16.212.193:41885)
I20240503 08:09:33.633468 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:33.633602 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:33.633638 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:33.633651 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723773633652 us; error 0 us; skew 500 ppm
I20240503 08:09:33.633687 21210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33594
I20240503 08:09:33.634389 17235 webserver.cc:460] Webserver started at http://127.16.212.194:33293/ using document root <none> and password file <none>
I20240503 08:09:33.634469 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:33.634541 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:33.634595 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:33.634894 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "30c7b79bde5b4714b62ea311700a3c20"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.635731 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:33.636161 21351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.636298 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.636338 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "30c7b79bde5b4714b62ea311700a3c20"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.636377 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:33.661419 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:33.661648 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:33.661826 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:33.662024 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:33.662051 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.662075 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:33.662104 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.664924 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:40851
I20240503 08:09:33.664999 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:40851 every 8 connection(s)
I20240503 08:09:33.665328 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:33.667716 21418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:33.668568 21421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.669042 21415 heartbeater.cc:347] Connected to a master server at 127.16.212.254:37363
I20240503 08:09:33.669106 21415 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:33.669253 21415 heartbeater.cc:511] Master 127.16.212.254:37363 requested a full tablet report, sending...
I20240503 08:09:33.669485 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:33.669638 21210 ts_manager.cc:194] Registered new tserver with Master: 30c7b79bde5b4714b62ea311700a3c20 (127.16.212.194:40851)
I20240503 08:09:33.670001 21210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33596
W20240503 08:09:33.670225 21419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:33.670385 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:33.670423 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:33.670447 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723773670446 us; error 0 us; skew 500 ppm
I20240503 08:09:33.671204 17235 webserver.cc:460] Webserver started at http://127.16.212.195:40685/ using document root <none> and password file <none>
I20240503 08:09:33.671283 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:33.671346 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:33.671399 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:33.671710 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "deb84c8969ff4316a2aae9317c84c83d"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.672600 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:33.673049 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.673221 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.673266 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "deb84c8969ff4316a2aae9317c84c83d"
format_stamp: "Formatted at 2024-05-03 08:09:33 on dist-test-slave-zl2t"
I20240503 08:09:33.673327 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:33.693009 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:33.693295 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:33.693480 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:33.693694 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:33.693722 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.693758 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:33.693784 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:33.696929 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:41357
I20240503 08:09:33.697011 21489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:41357 every 8 connection(s)
I20240503 08:09:33.700132 21490 heartbeater.cc:347] Connected to a master server at 127.16.212.254:37363
I20240503 08:09:33.700193 21490 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:33.700291 21490 heartbeater.cc:511] Master 127.16.212.254:37363 requested a full tablet report, sending...
I20240503 08:09:33.700553 21210 ts_manager.cc:194] Registered new tserver with Master: deb84c8969ff4316a2aae9317c84c83d (127.16.212.195:41357)
I20240503 08:09:33.700917 21210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33598
I20240503 08:09:33.701506 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004388525s
I20240503 08:09:33.701857 17235 test_util.cc:268] Using random seed: 1030446817
I20240503 08:09:33.706249 21210 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33610:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:33.706401 21210 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:33.711290 21379 tablet_service.cc:1467] Processing CreateTablet for tablet fd948ae1db24423d90ce00bafe70aad3 (DEFAULT_TABLE table=test-workload [id=0f43221885d8424a8c25cf7460cad0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:33.711309 21454 tablet_service.cc:1467] Processing CreateTablet for tablet fd948ae1db24423d90ce00bafe70aad3 (DEFAULT_TABLE table=test-workload [id=0f43221885d8424a8c25cf7460cad0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:33.711283 21303 tablet_service.cc:1467] Processing CreateTablet for tablet fd948ae1db24423d90ce00bafe70aad3 (DEFAULT_TABLE table=test-workload [id=0f43221885d8424a8c25cf7460cad0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:33.711447 21379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd948ae1db24423d90ce00bafe70aad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:33.711481 21303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd948ae1db24423d90ce00bafe70aad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:33.711481 21454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd948ae1db24423d90ce00bafe70aad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:33.712652 21505 tablet_bootstrap.cc:492] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: Bootstrap starting.
I20240503 08:09:33.712817 21506 tablet_bootstrap.cc:492] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d: Bootstrap starting.
I20240503 08:09:33.713243 21506 tablet_bootstrap.cc:654] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:33.713253 21505 tablet_bootstrap.cc:654] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:33.713791 21509 tablet_bootstrap.cc:492] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Bootstrap starting.
I20240503 08:09:33.713917 21506 tablet_bootstrap.cc:492] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d: No bootstrap required, opened a new log
I20240503 08:09:33.713972 21506 ts_tablet_manager.cc:1367] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:33.714135 21506 raft_consensus.cc:348] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.714226 21506 raft_consensus.cc:374] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:33.714249 21506 raft_consensus.cc:729] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb84c8969ff4316a2aae9317c84c83d, State: Initialized, Role: FOLLOWER
I20240503 08:09:33.714251 21509 tablet_bootstrap.cc:654] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:33.714457 21505 tablet_bootstrap.cc:492] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: No bootstrap required, opened a new log
I20240503 08:09:33.714505 21505 ts_tablet_manager.cc:1367] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:33.714682 21505 raft_consensus.cc:348] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.714746 21505 raft_consensus.cc:374] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:33.714771 21505 raft_consensus.cc:729] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c7b79bde5b4714b62ea311700a3c20, State: Initialized, Role: FOLLOWER
I20240503 08:09:33.714895 21509 tablet_bootstrap.cc:492] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: No bootstrap required, opened a new log
I20240503 08:09:33.714951 21509 ts_tablet_manager.cc:1367] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:33.714869 21505 consensus_queue.cc:260] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.714934 21506 consensus_queue.cc:260] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.715123 21509 raft_consensus.cc:348] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.715152 21505 ts_tablet_manager.cc:1398] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:33.715209 21506 ts_tablet_manager.cc:1398] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:33.715224 21490 heartbeater.cc:503] Master 127.16.212.254:37363 was elected leader, sending a full tablet report...
I20240503 08:09:33.715230 21509 raft_consensus.cc:374] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:33.715294 21415 heartbeater.cc:503] Master 127.16.212.254:37363 was elected leader, sending a full tablet report...
I20240503 08:09:33.715409 21509 raft_consensus.cc:729] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567ca14f62514273b683be3845f21950, State: Initialized, Role: FOLLOWER
I20240503 08:09:33.715487 21509 consensus_queue.cc:260] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.715653 21509 ts_tablet_manager.cc:1398] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:33.715750 21339 heartbeater.cc:503] Master 127.16.212.254:37363 was elected leader, sending a full tablet report...
I20240503 08:09:33.785308 21511 raft_consensus.cc:482] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:33.785415 21511 raft_consensus.cc:504] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.785579 21511 leader_election.cc:288] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 567ca14f62514273b683be3845f21950 (127.16.212.193:41885), deb84c8969ff4316a2aae9317c84c83d (127.16.212.195:41357)
I20240503 08:09:33.788976 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3" candidate_uuid: "30c7b79bde5b4714b62ea311700a3c20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb84c8969ff4316a2aae9317c84c83d" is_pre_election: true
I20240503 08:09:33.788976 21313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3" candidate_uuid: "30c7b79bde5b4714b62ea311700a3c20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567ca14f62514273b683be3845f21950" is_pre_election: true
I20240503 08:09:33.789124 21464 raft_consensus.cc:2454] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c7b79bde5b4714b62ea311700a3c20 in term 0.
I20240503 08:09:33.789132 21313 raft_consensus.cc:2454] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30c7b79bde5b4714b62ea311700a3c20 in term 0.
I20240503 08:09:33.789309 21354 leader_election.cc:302] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c7b79bde5b4714b62ea311700a3c20, 567ca14f62514273b683be3845f21950; no voters:
I20240503 08:09:33.789412 21511 raft_consensus.cc:2781] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:33.789477 21511 raft_consensus.cc:482] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:33.789492 21511 raft_consensus.cc:3037] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:33.790127 21511 raft_consensus.cc:504] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.790248 21511 leader_election.cc:288] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [CANDIDATE]: Term 1 election: Requested vote from peers 567ca14f62514273b683be3845f21950 (127.16.212.193:41885), deb84c8969ff4316a2aae9317c84c83d (127.16.212.195:41357)
I20240503 08:09:33.790408 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3" candidate_uuid: "30c7b79bde5b4714b62ea311700a3c20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb84c8969ff4316a2aae9317c84c83d"
I20240503 08:09:33.790443 21313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3" candidate_uuid: "30c7b79bde5b4714b62ea311700a3c20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567ca14f62514273b683be3845f21950"
I20240503 08:09:33.790477 21464 raft_consensus.cc:3037] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:33.790491 21313 raft_consensus.cc:3037] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:33.791131 21313 raft_consensus.cc:2454] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c7b79bde5b4714b62ea311700a3c20 in term 1.
I20240503 08:09:33.791193 21464 raft_consensus.cc:2454] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30c7b79bde5b4714b62ea311700a3c20 in term 1.
I20240503 08:09:33.791363 21352 leader_election.cc:302] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30c7b79bde5b4714b62ea311700a3c20, deb84c8969ff4316a2aae9317c84c83d; no voters:
I20240503 08:09:33.791451 21511 raft_consensus.cc:2781] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:33.791577 21511 raft_consensus.cc:686] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 LEADER]: Becoming Leader. State: Replica: 30c7b79bde5b4714b62ea311700a3c20, State: Running, Role: LEADER
I20240503 08:09:33.791620 21511 consensus_queue.cc:237] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:33.792158 21208 catalog_manager.cc:5484] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30c7b79bde5b4714b62ea311700a3c20 (127.16.212.194). New cstate: current_term: 1 leader_uuid: "30c7b79bde5b4714b62ea311700a3c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:33.810283 21464 raft_consensus.cc:1261] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 1 FOLLOWER]: Refusing update from remote peer 30c7b79bde5b4714b62ea311700a3c20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:33.810310 21313 raft_consensus.cc:1261] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 1 FOLLOWER]: Refusing update from remote peer 30c7b79bde5b4714b62ea311700a3c20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:33.810626 21515 consensus_queue.cc:1035] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:33.810746 21511 consensus_queue.cc:1035] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:44.319202 21511 raft_consensus.cc:979] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: : Instructing follower 567ca14f62514273b683be3845f21950 to start an election
I20240503 08:09:44.319281 21511 raft_consensus.cc:1067] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 LEADER]: Signalling peer 567ca14f62514273b683be3845f21950 to start an election
I20240503 08:09:44.319460 21313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3"
dest_uuid: "567ca14f62514273b683be3845f21950"
from {username='slave'} at 127.0.0.1:53616
I20240503 08:09:44.319556 21313 raft_consensus.cc:482] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:09:44.319587 21313 raft_consensus.cc:3037] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:44.320225 21313 raft_consensus.cc:504] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:44.320395 21313 leader_election.cc:288] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [CANDIDATE]: Term 2 election: Requested vote from peers 30c7b79bde5b4714b62ea311700a3c20 (127.16.212.194:40851), deb84c8969ff4316a2aae9317c84c83d (127.16.212.195:41357)
I20240503 08:09:44.323009 21464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3" candidate_uuid: "567ca14f62514273b683be3845f21950" candidate_term: 2 candidate_status { last_received { term: 1 index: 13535 } } ignore_live_leader: true dest_uuid: "deb84c8969ff4316a2aae9317c84c83d"
I20240503 08:09:44.323055 21389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd948ae1db24423d90ce00bafe70aad3" candidate_uuid: "567ca14f62514273b683be3845f21950" candidate_term: 2 candidate_status { last_received { term: 1 index: 13535 } } ignore_live_leader: true dest_uuid: "30c7b79bde5b4714b62ea311700a3c20"
I20240503 08:09:44.323100 21464 raft_consensus.cc:3037] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:44.323114 21389 raft_consensus.cc:3032] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:09:44.323133 21389 raft_consensus.cc:729] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30c7b79bde5b4714b62ea311700a3c20, State: Running, Role: LEADER
I20240503 08:09:44.323194 21389 consensus_queue.cc:260] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13535, Committed index: 13535, Last appended: 1.13535, Last appended by leader: 13535, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:44.323287 21389 raft_consensus.cc:3037] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:09:44.323783 21464 raft_consensus.cc:2454] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567ca14f62514273b683be3845f21950 in term 2.
I20240503 08:09:44.323873 21389 raft_consensus.cc:2454] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567ca14f62514273b683be3845f21950 in term 2.
I20240503 08:09:44.323985 21276 leader_election.cc:302] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567ca14f62514273b683be3845f21950, deb84c8969ff4316a2aae9317c84c83d; no voters:
I20240503 08:09:44.324179 21560 raft_consensus.cc:2781] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:09:44.324292 21560 raft_consensus.cc:686] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 2 LEADER]: Becoming Leader. State: Replica: 567ca14f62514273b683be3845f21950, State: Running, Role: LEADER
I20240503 08:09:44.324354 21560 consensus_queue.cc:237] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13535, Committed index: 13535, Last appended: 1.13535, Last appended by leader: 13535, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } }
I20240503 08:09:44.324774 21207 catalog_manager.cc:5484] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 reported cstate change: term changed from 1 to 2, leader changed from 30c7b79bde5b4714b62ea311700a3c20 (127.16.212.194) to 567ca14f62514273b683be3845f21950 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "567ca14f62514273b683be3845f21950" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567ca14f62514273b683be3845f21950" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30c7b79bde5b4714b62ea311700a3c20" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "deb84c8969ff4316a2aae9317c84c83d" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 41357 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:44.333072 17235 tablet_replica.cc:330] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: stopping tablet replica
I20240503 08:09:44.333149 17235 raft_consensus.cc:2229] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:09:44.333212 17235 pending_rounds.cc:62] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Trying to abort 1 pending ops.
I20240503 08:09:44.333235 17235 pending_rounds.cc:68] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Aborting op as it isn't in flight: id { term: 2 index: 13536 } timestamp: 7023508620593078272 op_type: NO_OP noop_request { }
I20240503 08:09:44.333294 17235 raft_consensus.cc:2866] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:09:44.333323 17235 raft_consensus.cc:2258] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:44.339113 17235 ts_tablet_manager.cc:1875] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:09:44.340991 17235 ts_tablet_manager.cc:1888] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13536
I20240503 08:09:44.341074 17235 log.cc:1196] T fd948ae1db24423d90ce00bafe70aad3 P 567ca14f62514273b683be3845f21950: Deleting WAL directory at /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/fd948ae1db24423d90ce00bafe70aad3
I20240503 08:09:44.350577 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:09:44.353564 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:44.365756 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:09:44.366827 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:09:44.369985 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:44.370128 17235 tablet_replica.cc:330] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20: stopping tablet replica
I20240503 08:09:44.370191 17235 raft_consensus.cc:2229] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:44.370239 17235 raft_consensus.cc:2258] T fd948ae1db24423d90ce00bafe70aad3 P 30c7b79bde5b4714b62ea311700a3c20 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:44.383584 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:09:44.384584 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:09:44.387380 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:44.387487 17235 tablet_replica.cc:330] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d: stopping tablet replica
I20240503 08:09:44.387537 17235 raft_consensus.cc:2229] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:44.387599 17235 raft_consensus.cc:2258] T fd948ae1db24423d90ce00bafe70aad3 P deb84c8969ff4316a2aae9317c84c83d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:44.410568 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:09:44.411401 17235 master.cc:537] Master@127.16.212.254:37363 shutting down...
I20240503 08:09:44.413463 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:44.413532 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:44.413551 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 67f13766f1944a74a1c865cba4b30c19: stopping tablet replica
I20240503 08:09:44.425148 17235 master.cc:552] Master@127.16.212.254:37363 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletNotRunning (10920 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletNotFound
I20240503 08:09:44.439555 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:43455
I20240503 08:09:44.439718 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:44.441803 21568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:44.442432 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:44.443406 21569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.444120 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:44.444278 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:44.444309 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:44.444334 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723784444333 us; error 0 us; skew 500 ppm
I20240503 08:09:44.445027 17235 webserver.cc:460] Webserver started at http://127.16.212.254:37615/ using document root <none> and password file <none>
I20240503 08:09:44.445118 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:44.445173 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:44.445226 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:44.445487 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "ff80a01959a84b2aa82691847ad94dc9"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.446380 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:44.446882 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.447021 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.447068 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "ff80a01959a84b2aa82691847ad94dc9"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.447121 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:44.459678 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:44.459868 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:44.462234 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:43455
I20240503 08:09:44.464531 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:43455 every 8 connection(s)
I20240503 08:09:44.464696 21629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:44.465683 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9: Bootstrap starting.
I20240503 08:09:44.465942 21629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:44.466450 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9: No bootstrap required, opened a new log
I20240503 08:09:44.466558 21629 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a01959a84b2aa82691847ad94dc9" member_type: VOTER }
I20240503 08:09:44.466636 21629 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:44.466661 21629 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff80a01959a84b2aa82691847ad94dc9, State: Initialized, Role: FOLLOWER
I20240503 08:09:44.466713 21629 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a01959a84b2aa82691847ad94dc9" member_type: VOTER }
I20240503 08:09:44.466754 21629 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:44.466775 21629 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:44.466795 21629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:44.467247 21629 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a01959a84b2aa82691847ad94dc9" member_type: VOTER }
I20240503 08:09:44.467298 21629 leader_election.cc:302] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff80a01959a84b2aa82691847ad94dc9; no voters:
I20240503 08:09:44.467396 21629 leader_election.cc:288] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:44.467456 21632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:44.467530 21629 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:44.467604 21632 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 1 LEADER]: Becoming Leader. State: Replica: ff80a01959a84b2aa82691847ad94dc9, State: Running, Role: LEADER
I20240503 08:09:44.467676 21632 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a01959a84b2aa82691847ad94dc9" member_type: VOTER }
I20240503 08:09:44.467844 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff80a01959a84b2aa82691847ad94dc9. Latest consensus state: current_term: 1 leader_uuid: "ff80a01959a84b2aa82691847ad94dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a01959a84b2aa82691847ad94dc9" member_type: VOTER } }
I20240503 08:09:44.467859 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff80a01959a84b2aa82691847ad94dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff80a01959a84b2aa82691847ad94dc9" member_type: VOTER } }
I20240503 08:09:44.467990 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:44.468094 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:44.468307 21641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:44.468438 21641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:44.468938 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:44.469044 21641 catalog_manager.cc:1347] Generated new cluster ID: d9d7453b21ec4e02ba2629309faa41dd
I20240503 08:09:44.469086 21641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:44.474635 21641 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:44.475018 21641 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:44.481882 21641 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9: Generated new TSK 0
I20240503 08:09:44.481976 21641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:44.484545 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:44.486862 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.487536 17235 server_base.cc:1034] running on GCE node
W20240503 08:09:44.488387 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:44.489024 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.489176 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:44.489215 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:44.489235 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723784489235 us; error 0 us; skew 500 ppm
I20240503 08:09:44.489874 17235 webserver.cc:460] Webserver started at http://127.16.212.193:35363/ using document root <none> and password file <none>
I20240503 08:09:44.489959 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:44.490006 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:44.490051 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:44.490322 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "ecddc93551dc4a6591bef3386dc8c216"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.491247 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:44.491720 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.491863 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.491906 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "ecddc93551dc4a6591bef3386dc8c216"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.491959 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:44.498560 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:44.498744 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:44.498869 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:44.499037 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:44.499063 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.499094 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:44.499121 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.502027 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:35041
I20240503 08:09:44.502565 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:09:44.502960 21721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:35041 every 8 connection(s)
W20240503 08:09:44.504694 21724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:44.505549 21725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.506306 21722 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43455
W20240503 08:09:44.506316 21727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.506393 21722 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:44.506492 21722 heartbeater.cc:511] Master 127.16.212.254:43455 requested a full tablet report, sending...
I20240503 08:09:44.506736 21593 ts_manager.cc:194] Registered new tserver with Master: ecddc93551dc4a6591bef3386dc8c216 (127.16.212.193:35041)
I20240503 08:09:44.506999 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:44.507097 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:44.507126 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:44.507150 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723784507150 us; error 0 us; skew 500 ppm
I20240503 08:09:44.507404 21593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49490
I20240503 08:09:44.507864 17235 webserver.cc:460] Webserver started at http://127.16.212.194:42927/ using document root <none> and password file <none>
I20240503 08:09:44.507944 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:44.507998 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:44.508055 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:44.508322 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "1fcdcb551d65407b9be708663a96f775"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.509141 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:44.509554 21734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.509685 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.509727 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "1fcdcb551d65407b9be708663a96f775"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.509778 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:44.520535 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:44.520753 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:44.520892 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:44.521081 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:44.521108 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.521142 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:44.521173 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.523963 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:43869
I20240503 08:09:44.524008 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:43869 every 8 connection(s)
I20240503 08:09:44.524318 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:44.526384 21801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.527204 21798 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43455
I20240503 08:09:44.527263 21798 heartbeater.cc:464] Registering TS with master...
W20240503 08:09:44.527261 21802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.527395 21798 heartbeater.cc:511] Master 127.16.212.254:43455 requested a full tablet report, sending...
I20240503 08:09:44.527644 21593 ts_manager.cc:194] Registered new tserver with Master: 1fcdcb551d65407b9be708663a96f775 (127.16.212.194:43869)
W20240503 08:09:44.527858 21804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:44.528084 21593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49504
I20240503 08:09:44.528656 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:44.528749 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:44.528774 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:44.528786 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723784528787 us; error 0 us; skew 500 ppm
I20240503 08:09:44.529363 17235 webserver.cc:460] Webserver started at http://127.16.212.195:40007/ using document root <none> and password file <none>
I20240503 08:09:44.529428 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:44.529469 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:44.529503 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:44.529774 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "cce500b48c4f4ebb88acc004094866f6"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.530550 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:44.531023 21809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.531148 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.531198 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "cce500b48c4f4ebb88acc004094866f6"
format_stamp: "Formatted at 2024-05-03 08:09:44 on dist-test-slave-zl2t"
I20240503 08:09:44.531234 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:44.540971 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:44.541182 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:44.541308 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:44.541486 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:44.541517 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.541538 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:44.541564 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.544651 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:39613
I20240503 08:09:44.544711 21872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:39613 every 8 connection(s)
I20240503 08:09:44.547793 21873 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43455
I20240503 08:09:44.547854 21873 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:44.547967 21873 heartbeater.cc:511] Master 127.16.212.254:43455 requested a full tablet report, sending...
I20240503 08:09:44.548199 21593 ts_manager.cc:194] Registered new tserver with Master: cce500b48c4f4ebb88acc004094866f6 (127.16.212.195:39613)
I20240503 08:09:44.548550 21593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49512
I20240503 08:09:44.549214 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004387808s
I20240503 08:09:44.549556 17235 test_util.cc:268] Using random seed: 1041294516
I20240503 08:09:44.553987 21593 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49528:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:44.554138 21593 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:44.558496 21837 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1aabd423e2400eb6ab6e4c70afc8b2 (DEFAULT_TABLE table=test-workload [id=662cf6694e3d4d3294cf09d13275836f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:44.558691 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1aabd423e2400eb6ab6e4c70afc8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:44.558960 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1aabd423e2400eb6ab6e4c70afc8b2 (DEFAULT_TABLE table=test-workload [id=662cf6694e3d4d3294cf09d13275836f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:44.559089 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1aabd423e2400eb6ab6e4c70afc8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:44.559573 21762 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1aabd423e2400eb6ab6e4c70afc8b2 (DEFAULT_TABLE table=test-workload [id=662cf6694e3d4d3294cf09d13275836f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:44.559679 21762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1aabd423e2400eb6ab6e4c70afc8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:44.560281 21888 tablet_bootstrap.cc:492] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6: Bootstrap starting.
I20240503 08:09:44.560462 21889 tablet_bootstrap.cc:492] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216: Bootstrap starting.
I20240503 08:09:44.560930 21888 tablet_bootstrap.cc:654] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:44.560966 21890 tablet_bootstrap.cc:492] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775: Bootstrap starting.
I20240503 08:09:44.560936 21889 tablet_bootstrap.cc:654] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:44.561338 21890 tablet_bootstrap.cc:654] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:44.561676 21888 tablet_bootstrap.cc:492] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6: No bootstrap required, opened a new log
I20240503 08:09:44.561735 21888 ts_tablet_manager.cc:1367] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:44.561909 21890 tablet_bootstrap.cc:492] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775: No bootstrap required, opened a new log
I20240503 08:09:44.561889 21888 raft_consensus.cc:348] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.561951 21888 raft_consensus.cc:374] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:44.561986 21888 raft_consensus.cc:729] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce500b48c4f4ebb88acc004094866f6, State: Initialized, Role: FOLLOWER
I20240503 08:09:44.561952 21890 ts_tablet_manager.cc:1367] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:44.562047 21888 consensus_queue.cc:260] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.562150 21890 raft_consensus.cc:348] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.562209 21890 raft_consensus.cc:374] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:44.562232 21890 raft_consensus.cc:729] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcdcb551d65407b9be708663a96f775, State: Initialized, Role: FOLLOWER
I20240503 08:09:44.562255 21873 heartbeater.cc:503] Master 127.16.212.254:43455 was elected leader, sending a full tablet report...
I20240503 08:09:44.562233 21888 ts_tablet_manager.cc:1398] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:09:44.562320 21890 consensus_queue.cc:260] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.562393 21889 tablet_bootstrap.cc:492] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216: No bootstrap required, opened a new log
I20240503 08:09:44.562430 21889 ts_tablet_manager.cc:1367] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:09:44.562481 21890 ts_tablet_manager.cc:1398] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:44.562577 21889 raft_consensus.cc:348] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.562639 21798 heartbeater.cc:503] Master 127.16.212.254:43455 was elected leader, sending a full tablet report...
I20240503 08:09:44.562821 21889 raft_consensus.cc:374] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:44.562872 21889 raft_consensus.cc:729] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecddc93551dc4a6591bef3386dc8c216, State: Initialized, Role: FOLLOWER
I20240503 08:09:44.563009 21889 consensus_queue.cc:260] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.563202 21889 ts_tablet_manager.cc:1398] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:44.563247 21722 heartbeater.cc:503] Master 127.16.212.254:43455 was elected leader, sending a full tablet report...
I20240503 08:09:44.990224 21896 raft_consensus.cc:482] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:44.990350 21896 raft_consensus.cc:504] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.990592 21896 leader_election.cc:288] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce500b48c4f4ebb88acc004094866f6 (127.16.212.195:39613), 1fcdcb551d65407b9be708663a96f775 (127.16.212.194:43869)
I20240503 08:09:44.993227 21895 raft_consensus.cc:482] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:44.993228 21847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1aabd423e2400eb6ab6e4c70afc8b2" candidate_uuid: "ecddc93551dc4a6591bef3386dc8c216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce500b48c4f4ebb88acc004094866f6" is_pre_election: true
I20240503 08:09:44.993336 21895 raft_consensus.cc:504] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.993371 21847 raft_consensus.cc:2454] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecddc93551dc4a6591bef3386dc8c216 in term 0.
I20240503 08:09:44.993502 21895 leader_election.cc:288] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce500b48c4f4ebb88acc004094866f6 (127.16.212.195:39613), ecddc93551dc4a6591bef3386dc8c216 (127.16.212.193:35041)
I20240503 08:09:44.993561 21661 leader_election.cc:302] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cce500b48c4f4ebb88acc004094866f6, ecddc93551dc4a6591bef3386dc8c216; no voters:
I20240503 08:09:44.993561 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1aabd423e2400eb6ab6e4c70afc8b2" candidate_uuid: "ecddc93551dc4a6591bef3386dc8c216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcdcb551d65407b9be708663a96f775" is_pre_election: true
I20240503 08:09:44.993865 21772 raft_consensus.cc:2454] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecddc93551dc4a6591bef3386dc8c216 in term 0.
I20240503 08:09:44.993973 21896 raft_consensus.cc:2781] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:44.994078 21896 raft_consensus.cc:482] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:44.994099 21896 raft_consensus.cc:3037] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:44.994964 21896 raft_consensus.cc:504] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.995193 21847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1aabd423e2400eb6ab6e4c70afc8b2" candidate_uuid: "ecddc93551dc4a6591bef3386dc8c216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce500b48c4f4ebb88acc004094866f6"
I20240503 08:09:44.995277 21772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1aabd423e2400eb6ab6e4c70afc8b2" candidate_uuid: "ecddc93551dc4a6591bef3386dc8c216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fcdcb551d65407b9be708663a96f775"
I20240503 08:09:44.995260 21847 raft_consensus.cc:3037] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:44.995347 21772 raft_consensus.cc:3037] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:44.996050 21847 raft_consensus.cc:2454] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecddc93551dc4a6591bef3386dc8c216 in term 1.
I20240503 08:09:44.996050 21772 raft_consensus.cc:2454] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecddc93551dc4a6591bef3386dc8c216 in term 1.
I20240503 08:09:44.996587 21847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1aabd423e2400eb6ab6e4c70afc8b2" candidate_uuid: "1fcdcb551d65407b9be708663a96f775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce500b48c4f4ebb88acc004094866f6" is_pre_election: true
I20240503 08:09:44.996661 21696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1aabd423e2400eb6ab6e4c70afc8b2" candidate_uuid: "1fcdcb551d65407b9be708663a96f775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecddc93551dc4a6591bef3386dc8c216" is_pre_election: true
I20240503 08:09:44.996672 21847 raft_consensus.cc:2379] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcdcb551d65407b9be708663a96f775 in current term 1: Already voted for candidate ecddc93551dc4a6591bef3386dc8c216 in this term.
I20240503 08:09:44.996605 21661 leader_election.cc:302] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fcdcb551d65407b9be708663a96f775, ecddc93551dc4a6591bef3386dc8c216; no voters:
I20240503 08:09:44.996729 21696 raft_consensus.cc:2379] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1fcdcb551d65407b9be708663a96f775 in current term 1: Already voted for candidate ecddc93551dc4a6591bef3386dc8c216 in this term.
I20240503 08:09:44.996734 21896 leader_election.cc:288] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [CANDIDATE]: Term 1 election: Requested vote from peers cce500b48c4f4ebb88acc004094866f6 (127.16.212.195:39613), 1fcdcb551d65407b9be708663a96f775 (127.16.212.194:43869)
I20240503 08:09:44.996958 21896 raft_consensus.cc:2781] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:44.996989 21896 raft_consensus.cc:686] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 1 LEADER]: Becoming Leader. State: Replica: ecddc93551dc4a6591bef3386dc8c216, State: Running, Role: LEADER
I20240503 08:09:44.997031 21896 consensus_queue.cc:237] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } }
I20240503 08:09:44.997035 21737 leader_election.cc:302] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1fcdcb551d65407b9be708663a96f775; no voters: cce500b48c4f4ebb88acc004094866f6, ecddc93551dc4a6591bef3386dc8c216
I20240503 08:09:44.997179 21895 raft_consensus.cc:2726] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:09:44.997586 21592 catalog_manager.cc:5484] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecddc93551dc4a6591bef3386dc8c216 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "ecddc93551dc4a6591bef3386dc8c216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecddc93551dc4a6591bef3386dc8c216" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:45.118698 21847 raft_consensus.cc:1261] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 1 FOLLOWER]: Refusing update from remote peer ecddc93551dc4a6591bef3386dc8c216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:45.118719 21772 raft_consensus.cc:1261] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 1 FOLLOWER]: Refusing update from remote peer ecddc93551dc4a6591bef3386dc8c216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:45.118963 21899 consensus_queue.cc:1035] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce500b48c4f4ebb88acc004094866f6" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:45.119067 21896 consensus_queue.cc:1035] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fcdcb551d65407b9be708663a96f775" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:55.115121 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:09:55.118573 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:55.118747 17235 tablet_replica.cc:330] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216: stopping tablet replica
I20240503 08:09:55.118816 17235 raft_consensus.cc:2229] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:55.118887 17235 raft_consensus.cc:2258] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P ecddc93551dc4a6591bef3386dc8c216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:55.133482 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:09:55.134349 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:09:55.137826 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:55.137948 17235 tablet_replica.cc:330] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775: stopping tablet replica
I20240503 08:09:55.138010 17235 raft_consensus.cc:2229] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:55.138077 17235 raft_consensus.cc:2258] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P 1fcdcb551d65407b9be708663a96f775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:55.159289 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:09:55.160295 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:09:55.163729 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:09:55.163870 17235 tablet_replica.cc:330] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6: stopping tablet replica
I20240503 08:09:55.163933 17235 raft_consensus.cc:2229] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:09:55.164011 17235 raft_consensus.cc:2258] T 5e1aabd423e2400eb6ab6e4c70afc8b2 P cce500b48c4f4ebb88acc004094866f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:55.167444 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:09:55.168298 17235 master.cc:537] Master@127.16.212.254:43455 shutting down...
I20240503 08:09:55.170899 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:09:55.170962 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:09:55.170989 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P ff80a01959a84b2aa82691847ad94dc9: stopping tablet replica
I20240503 08:09:55.183341 17235 master.cc:552] Master@127.16.212.254:43455 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletNotFound (10763 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientGetMetadata
I20240503 08:09:55.202692 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:42193
I20240503 08:09:55.202850 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:55.205330 21942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:55.205932 17235 server_base.cc:1034] running on GCE node
W20240503 08:09:55.206836 21945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.207330 21943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:55.207492 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:55.207525 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:55.207537 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723795207537 us; error 0 us; skew 500 ppm
I20240503 08:09:55.208173 17235 webserver.cc:460] Webserver started at http://127.16.212.254:41411/ using document root <none> and password file <none>
I20240503 08:09:55.208241 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:55.208315 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:55.208343 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:55.208591 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.209492 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.209960 21950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.210126 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.210171 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.210223 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:55.227210 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:55.227402 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:55.229805 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:42193
I20240503 08:09:55.233121 22003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:55.239436 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:42193 every 8 connection(s)
I20240503 08:09:55.239677 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf: Bootstrap starting.
I20240503 08:09:55.239990 22003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:55.240598 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf: No bootstrap required, opened a new log
I20240503 08:09:55.240712 22003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" member_type: VOTER }
I20240503 08:09:55.240763 22003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:55.240777 22003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871e18cd5ef54b1c9ece2c92ebfca4bf, State: Initialized, Role: FOLLOWER
I20240503 08:09:55.240813 22003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" member_type: VOTER }
I20240503 08:09:55.240833 22003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:09:55.240873 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:09:55.240914 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:55.241434 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" member_type: VOTER }
I20240503 08:09:55.241490 22003 leader_election.cc:302] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 871e18cd5ef54b1c9ece2c92ebfca4bf; no voters:
I20240503 08:09:55.241593 22003 leader_election.cc:288] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:09:55.241700 22006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:55.241741 22003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:09:55.241865 22006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 1 LEADER]: Becoming Leader. State: Replica: 871e18cd5ef54b1c9ece2c92ebfca4bf, State: Running, Role: LEADER
I20240503 08:09:55.241923 22006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" member_type: VOTER }
I20240503 08:09:55.242110 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" member_type: VOTER } }
I20240503 08:09:55.242192 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:55.242420 22008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 871e18cd5ef54b1c9ece2c92ebfca4bf. Latest consensus state: current_term: 1 leader_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "871e18cd5ef54b1c9ece2c92ebfca4bf" member_type: VOTER } }
I20240503 08:09:55.242501 22008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [sys.catalog]: This master's current role is: LEADER
I20240503 08:09:55.242545 22012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:09:55.242713 22012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:09:55.243110 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:09:55.243305 22012 catalog_manager.cc:1347] Generated new cluster ID: 9c6be390559c4c7abef765d66c20e7c7
I20240503 08:09:55.243341 22012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:09:55.264606 22012 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:09:55.265058 22012 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:09:55.269542 22012 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf: Generated new TSK 0
I20240503 08:09:55.269624 22012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:09:55.275126 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:55.277316 22024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.277904 22025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.278759 22027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:55.279559 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:55.279676 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:55.279716 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:55.279729 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723795279730 us; error 0 us; skew 500 ppm
I20240503 08:09:55.280429 17235 webserver.cc:460] Webserver started at http://127.16.212.193:40001/ using document root <none> and password file <none>
I20240503 08:09:55.280526 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:55.280583 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:55.280633 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:55.280920 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "8cf661234f674a5db94b64ab1fff2283"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.281771 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.282189 22032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.282327 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.282373 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "8cf661234f674a5db94b64ab1fff2283"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.282424 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:55.289548 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:55.289733 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:55.289872 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:55.290058 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:55.290084 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.290117 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:55.290144 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.292770 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:35111
I20240503 08:09:55.292821 22095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:35111 every 8 connection(s)
I20240503 08:09:55.293129 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:55.294947 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.295570 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.296402 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:55.296521 22096 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42193
I20240503 08:09:55.296576 22096 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:55.296690 22096 heartbeater.cc:511] Master 127.16.212.254:42193 requested a full tablet report, sending...
I20240503 08:09:55.296957 21967 ts_manager.cc:194] Registered new tserver with Master: 8cf661234f674a5db94b64ab1fff2283 (127.16.212.193:35111)
I20240503 08:09:55.297220 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:55.297358 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:55.297389 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:55.297416 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723795297416 us; error 0 us; skew 500 ppm
I20240503 08:09:55.297699 21967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41144
I20240503 08:09:55.298146 17235 webserver.cc:460] Webserver started at http://127.16.212.194:36533/ using document root <none> and password file <none>
I20240503 08:09:55.298224 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:55.298275 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:55.298322 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:55.298591 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "44fd08c9f5c047249512675b9c3998aa"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.299511 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.299930 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.300081 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.300130 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "44fd08c9f5c047249512675b9c3998aa"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.300168 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:55.307844 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:55.308012 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:55.308130 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:55.308300 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:55.308327 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.308348 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:55.308375 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.311116 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:45717
I20240503 08:09:55.311211 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:45717 every 8 connection(s)
I20240503 08:09:55.311843 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:09:55.313990 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.314818 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:09:55.315522 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:09:55.315539 22172 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42193
I20240503 08:09:55.315661 22172 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:55.315786 22172 heartbeater.cc:511] Master 127.16.212.254:42193 requested a full tablet report, sending...
I20240503 08:09:55.316001 21967 ts_manager.cc:194] Registered new tserver with Master: 44fd08c9f5c047249512675b9c3998aa (127.16.212.194:45717)
I20240503 08:09:55.316236 17235 server_base.cc:1034] running on GCE node
I20240503 08:09:55.316320 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:09:55.316349 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:09:55.316373 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723795316374 us; error 0 us; skew 500 ppm
I20240503 08:09:55.316380 21967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41154
I20240503 08:09:55.317075 17235 webserver.cc:460] Webserver started at http://127.16.212.195:42439/ using document root <none> and password file <none>
I20240503 08:09:55.317154 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:09:55.317209 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:09:55.317260 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:09:55.317564 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "233ac3b8dd974294a7cc90aee7d2b5ef"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.318380 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:09:55.318861 22183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.318994 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.319034 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "233ac3b8dd974294a7cc90aee7d2b5ef"
format_stamp: "Formatted at 2024-05-03 08:09:55 on dist-test-slave-zl2t"
I20240503 08:09:55.319069 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:09:55.331363 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:09:55.331583 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:09:55.331730 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:09:55.331897 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:09:55.331925 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.331955 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:09:55.331976 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:09:55.335187 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:38219
I20240503 08:09:55.335224 22246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:38219 every 8 connection(s)
I20240503 08:09:55.338207 22247 heartbeater.cc:347] Connected to a master server at 127.16.212.254:42193
I20240503 08:09:55.338264 22247 heartbeater.cc:464] Registering TS with master...
I20240503 08:09:55.338366 22247 heartbeater.cc:511] Master 127.16.212.254:42193 requested a full tablet report, sending...
I20240503 08:09:55.338665 21967 ts_manager.cc:194] Registered new tserver with Master: 233ac3b8dd974294a7cc90aee7d2b5ef (127.16.212.195:38219)
I20240503 08:09:55.338738 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003374026s
I20240503 08:09:55.339143 17235 test_util.cc:268] Using random seed: 1052084103
I20240503 08:09:55.339155 21967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41156
I20240503 08:09:55.344261 21967 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41172:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:09:55.344410 21967 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:09:55.349764 22060 tablet_service.cc:1467] Processing CreateTablet for tablet 843298c232ec429cacada1f4fc49c1cd (DEFAULT_TABLE table=test-workload [id=aa3613296aea40fdadb35846efc85a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:55.349944 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843298c232ec429cacada1f4fc49c1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:55.350433 22211 tablet_service.cc:1467] Processing CreateTablet for tablet 843298c232ec429cacada1f4fc49c1cd (DEFAULT_TABLE table=test-workload [id=aa3613296aea40fdadb35846efc85a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:55.350549 22211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843298c232ec429cacada1f4fc49c1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:55.350775 22136 tablet_service.cc:1467] Processing CreateTablet for tablet 843298c232ec429cacada1f4fc49c1cd (DEFAULT_TABLE table=test-workload [id=aa3613296aea40fdadb35846efc85a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:09:55.350863 22136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843298c232ec429cacada1f4fc49c1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:09:55.351469 22262 tablet_bootstrap.cc:492] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283: Bootstrap starting.
I20240503 08:09:55.351877 22262 tablet_bootstrap.cc:654] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:55.352126 22263 tablet_bootstrap.cc:492] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef: Bootstrap starting.
I20240503 08:09:55.352502 22263 tablet_bootstrap.cc:654] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:55.352785 22264 tablet_bootstrap.cc:492] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: Bootstrap starting.
I20240503 08:09:55.353181 22263 tablet_bootstrap.cc:492] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef: No bootstrap required, opened a new log
I20240503 08:09:55.353216 22264 tablet_bootstrap.cc:654] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: Neither blocks nor log segments found. Creating new log.
I20240503 08:09:55.353246 22263 ts_tablet_manager.cc:1367] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.353423 22263 raft_consensus.cc:348] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.353592 22263 raft_consensus.cc:374] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:55.353636 22263 raft_consensus.cc:729] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233ac3b8dd974294a7cc90aee7d2b5ef, State: Initialized, Role: FOLLOWER
I20240503 08:09:55.353691 22263 consensus_queue.cc:260] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.353873 22263 ts_tablet_manager.cc:1398] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.353912 22262 tablet_bootstrap.cc:492] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283: No bootstrap required, opened a new log
I20240503 08:09:55.353953 22247 heartbeater.cc:503] Master 127.16.212.254:42193 was elected leader, sending a full tablet report...
I20240503 08:09:55.353960 22262 ts_tablet_manager.cc:1367] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 08:09:55.354032 22264 tablet_bootstrap.cc:492] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: No bootstrap required, opened a new log
I20240503 08:09:55.354074 22264 ts_tablet_manager.cc:1367] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.354125 22262 raft_consensus.cc:348] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.354207 22262 raft_consensus.cc:374] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:55.354218 22264 raft_consensus.cc:348] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.354310 22264 raft_consensus.cc:374] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:09:55.354277 22262 raft_consensus.cc:729] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf661234f674a5db94b64ab1fff2283, State: Initialized, Role: FOLLOWER
I20240503 08:09:55.354338 22264 raft_consensus.cc:729] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fd08c9f5c047249512675b9c3998aa, State: Initialized, Role: FOLLOWER
I20240503 08:09:55.354394 22262 consensus_queue.cc:260] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.354430 22264 consensus_queue.cc:260] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.354586 22262 ts_tablet_manager.cc:1398] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:09:55.354652 22096 heartbeater.cc:503] Master 127.16.212.254:42193 was elected leader, sending a full tablet report...
I20240503 08:09:55.354673 22264 ts_tablet_manager.cc:1398] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:09:55.354701 22172 heartbeater.cc:503] Master 127.16.212.254:42193 was elected leader, sending a full tablet report...
I20240503 08:09:55.418030 22270 raft_consensus.cc:482] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:09:55.418159 22270 raft_consensus.cc:504] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.418509 22270 leader_election.cc:288] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 233ac3b8dd974294a7cc90aee7d2b5ef (127.16.212.195:38219), 8cf661234f674a5db94b64ab1fff2283 (127.16.212.193:35111)
I20240503 08:09:55.421360 22221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd" candidate_uuid: "44fd08c9f5c047249512675b9c3998aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" is_pre_election: true
I20240503 08:09:55.421448 22221 raft_consensus.cc:2454] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fd08c9f5c047249512675b9c3998aa in term 0.
I20240503 08:09:55.421476 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd" candidate_uuid: "44fd08c9f5c047249512675b9c3998aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf661234f674a5db94b64ab1fff2283" is_pre_election: true
I20240503 08:09:55.421569 22070 raft_consensus.cc:2454] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fd08c9f5c047249512675b9c3998aa in term 0.
I20240503 08:09:55.421589 22112 leader_election.cc:302] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233ac3b8dd974294a7cc90aee7d2b5ef, 44fd08c9f5c047249512675b9c3998aa; no voters:
I20240503 08:09:55.421741 22270 raft_consensus.cc:2781] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:09:55.421787 22270 raft_consensus.cc:482] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:09:55.421800 22270 raft_consensus.cc:3037] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:55.422482 22270 raft_consensus.cc:504] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.422592 22270 leader_election.cc:288] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [CANDIDATE]: Term 1 election: Requested vote from peers 233ac3b8dd974294a7cc90aee7d2b5ef (127.16.212.195:38219), 8cf661234f674a5db94b64ab1fff2283 (127.16.212.193:35111)
I20240503 08:09:55.422775 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd" candidate_uuid: "44fd08c9f5c047249512675b9c3998aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf661234f674a5db94b64ab1fff2283"
I20240503 08:09:55.422780 22221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd" candidate_uuid: "44fd08c9f5c047249512675b9c3998aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef"
I20240503 08:09:55.422850 22221 raft_consensus.cc:3037] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:55.422853 22070 raft_consensus.cc:3037] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:09:55.423449 22221 raft_consensus.cc:2454] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fd08c9f5c047249512675b9c3998aa in term 1.
I20240503 08:09:55.423542 22070 raft_consensus.cc:2454] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fd08c9f5c047249512675b9c3998aa in term 1.
I20240503 08:09:55.423606 22112 leader_election.cc:302] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233ac3b8dd974294a7cc90aee7d2b5ef, 44fd08c9f5c047249512675b9c3998aa; no voters:
I20240503 08:09:55.423677 22270 raft_consensus.cc:2781] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:09:55.423820 22270 raft_consensus.cc:686] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 LEADER]: Becoming Leader. State: Replica: 44fd08c9f5c047249512675b9c3998aa, State: Running, Role: LEADER
I20240503 08:09:55.423864 22270 consensus_queue.cc:237] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:09:55.424388 21966 catalog_manager.cc:5484] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 44fd08c9f5c047249512675b9c3998aa (127.16.212.194). New cstate: current_term: 1 leader_uuid: "44fd08c9f5c047249512675b9c3998aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } health_report { overall_health: UNKNOWN } } }
I20240503 08:09:55.450202 22070 raft_consensus.cc:1261] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 1 FOLLOWER]: Refusing update from remote peer 44fd08c9f5c047249512675b9c3998aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:55.450240 22221 raft_consensus.cc:1261] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 1 FOLLOWER]: Refusing update from remote peer 44fd08c9f5c047249512675b9c3998aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:09:55.450523 22272 consensus_queue.cc:1035] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:09:55.450672 22270 consensus_queue.cc:1035] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:05.954692 22314 raft_consensus.cc:979] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: : Instructing follower 8cf661234f674a5db94b64ab1fff2283 to start an election
I20240503 08:10:05.954793 22314 raft_consensus.cc:1067] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 LEADER]: Signalling peer 8cf661234f674a5db94b64ab1fff2283 to start an election
I20240503 08:10:05.955109 22070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd"
dest_uuid: "8cf661234f674a5db94b64ab1fff2283"
from {username='slave'} at 127.0.0.1:50096
I20240503 08:10:05.955199 22070 raft_consensus.cc:482] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:10:05.955221 22070 raft_consensus.cc:3037] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:05.955896 22070 raft_consensus.cc:504] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:10:05.956059 22070 leader_election.cc:288] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [CANDIDATE]: Term 2 election: Requested vote from peers 44fd08c9f5c047249512675b9c3998aa (127.16.212.194:45717), 233ac3b8dd974294a7cc90aee7d2b5ef (127.16.212.195:38219)
I20240503 08:10:05.958771 22221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd" candidate_uuid: "8cf661234f674a5db94b64ab1fff2283" candidate_term: 2 candidate_status { last_received { term: 1 index: 13267 } } ignore_live_leader: true dest_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef"
I20240503 08:10:05.958771 22146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843298c232ec429cacada1f4fc49c1cd" candidate_uuid: "8cf661234f674a5db94b64ab1fff2283" candidate_term: 2 candidate_status { last_received { term: 1 index: 13267 } } ignore_live_leader: true dest_uuid: "44fd08c9f5c047249512675b9c3998aa"
I20240503 08:10:05.958906 22146 raft_consensus.cc:3032] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:10:05.958895 22221 raft_consensus.cc:3037] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:05.958977 22146 raft_consensus.cc:729] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44fd08c9f5c047249512675b9c3998aa, State: Running, Role: LEADER
I20240503 08:10:05.959067 22146 consensus_queue.cc:260] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13267, Committed index: 13267, Last appended: 1.13267, Last appended by leader: 13267, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:10:05.959182 22146 raft_consensus.cc:3037] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:05.959839 22221 raft_consensus.cc:2454] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cf661234f674a5db94b64ab1fff2283 in term 2.
I20240503 08:10:05.959839 22146 raft_consensus.cc:2454] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cf661234f674a5db94b64ab1fff2283 in term 2.
I20240503 08:10:05.960052 22035 leader_election.cc:302] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44fd08c9f5c047249512675b9c3998aa, 8cf661234f674a5db94b64ab1fff2283; no voters:
I20240503 08:10:05.960250 22320 raft_consensus.cc:2781] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:05.960374 22320 raft_consensus.cc:686] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 2 LEADER]: Becoming Leader. State: Replica: 8cf661234f674a5db94b64ab1fff2283, State: Running, Role: LEADER
I20240503 08:10:05.960423 22320 consensus_queue.cc:237] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13267, Committed index: 13267, Last appended: 1.13267, Last appended by leader: 13267, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } }
I20240503 08:10:05.960872 21959 catalog_manager.cc:5484] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 reported cstate change: term changed from 1 to 2, leader changed from 44fd08c9f5c047249512675b9c3998aa (127.16.212.194) to 8cf661234f674a5db94b64ab1fff2283 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "8cf661234f674a5db94b64ab1fff2283" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cf661234f674a5db94b64ab1fff2283" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 35111 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:05.971110 22221 raft_consensus.cc:1261] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 2 FOLLOWER]: Refusing update from remote peer 8cf661234f674a5db94b64ab1fff2283: Log matching property violated. Preceding OpId in replica: term: 1 index: 13267. Preceding OpId from leader: term: 2 index: 13269. (index mismatch)
I20240503 08:10:05.971127 22146 raft_consensus.cc:1261] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 2 FOLLOWER]: Refusing update from remote peer 8cf661234f674a5db94b64ab1fff2283: Log matching property violated. Preceding OpId in replica: term: 1 index: 13267. Preceding OpId from leader: term: 2 index: 13269. (index mismatch)
I20240503 08:10:05.971347 22321 consensus_queue.cc:1035] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44fd08c9f5c047249512675b9c3998aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13268, Last known committed idx: 13267, Time since last communication: 0.000s
I20240503 08:10:05.971423 22320 consensus_queue.cc:1035] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233ac3b8dd974294a7cc90aee7d2b5ef" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 38219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13268, Last known committed idx: 13267, Time since last communication: 0.000s
I20240503 08:10:05.973172 22330 mvcc.cc:205] Tried to move back new op lower bound from 7023508709256343552 to 7023508709214384128. Current Snapshot: MvccSnapshot[applied={T|T < 7023508709256343552}]
I20240503 08:10:05.977775 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:10:05.980260 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:05.980362 17235 tablet_replica.cc:330] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283: stopping tablet replica
I20240503 08:10:05.980412 17235 raft_consensus.cc:2229] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:05.980464 17235 raft_consensus.cc:2258] T 843298c232ec429cacada1f4fc49c1cd P 8cf661234f674a5db94b64ab1fff2283 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:05.995271 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:10:05.996168 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:10:05.999064 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:05.999195 17235 tablet_replica.cc:330] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa: stopping tablet replica
I20240503 08:10:05.999249 17235 raft_consensus.cc:2229] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:05.999300 17235 raft_consensus.cc:2258] T 843298c232ec429cacada1f4fc49c1cd P 44fd08c9f5c047249512675b9c3998aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:06.012771 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:10:06.013612 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:10:06.016826 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:06.016960 17235 tablet_replica.cc:330] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef: stopping tablet replica
I20240503 08:10:06.017030 17235 raft_consensus.cc:2229] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:06.017079 17235 raft_consensus.cc:2258] T 843298c232ec429cacada1f4fc49c1cd P 233ac3b8dd974294a7cc90aee7d2b5ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:06.040539 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:10:06.041543 17235 master.cc:537] Master@127.16.212.254:42193 shutting down...
I20240503 08:10:06.043896 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:06.043977 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:06.043995 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 871e18cd5ef54b1c9ece2c92ebfca4bf: stopping tablet replica
I20240503 08:10:06.056145 17235 master.cc:552] Master@127.16.212.254:42193 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientGetMetadata (10872 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock
I20240503 08:10:06.075248 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:39067
I20240503 08:10:06.075416 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:06.077404 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:06.078052 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:06.078774 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:06.079461 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:06.079566 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:06.079592 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:06.079604 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723806079604 us; error 0 us; skew 500 ppm
I20240503 08:10:06.080233 17235 webserver.cc:460] Webserver started at http://127.16.212.254:33827/ using document root <none> and password file <none>
I20240503 08:10:06.080300 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:06.080336 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:06.080370 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:06.080614 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "76876e6196f54e7380898b09f3687a26"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.081465 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:06.081849 22345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.081974 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.082017 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "76876e6196f54e7380898b09f3687a26"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.082070 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:06.099933 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:06.100163 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:06.102710 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:39067
I20240503 08:10:06.106904 22397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:39067 every 8 connection(s)
I20240503 08:10:06.107043 22398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:06.109243 22398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26: Bootstrap starting.
I20240503 08:10:06.109491 22398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:06.110003 22398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26: No bootstrap required, opened a new log
I20240503 08:10:06.110101 22398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876e6196f54e7380898b09f3687a26" member_type: VOTER }
I20240503 08:10:06.110150 22398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:06.110163 22398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76876e6196f54e7380898b09f3687a26, State: Initialized, Role: FOLLOWER
I20240503 08:10:06.110201 22398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876e6196f54e7380898b09f3687a26" member_type: VOTER }
I20240503 08:10:06.110241 22398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:06.110266 22398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:06.110291 22398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:06.110742 22398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876e6196f54e7380898b09f3687a26" member_type: VOTER }
I20240503 08:10:06.110796 22398 leader_election.cc:302] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76876e6196f54e7380898b09f3687a26; no voters:
I20240503 08:10:06.110888 22398 leader_election.cc:288] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:06.110960 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:06.111027 22398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:06.111111 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 1 LEADER]: Becoming Leader. State: Replica: 76876e6196f54e7380898b09f3687a26, State: Running, Role: LEADER
I20240503 08:10:06.111167 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876e6196f54e7380898b09f3687a26" member_type: VOTER }
I20240503 08:10:06.111382 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76876e6196f54e7380898b09f3687a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876e6196f54e7380898b09f3687a26" member_type: VOTER } }
I20240503 08:10:06.111392 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76876e6196f54e7380898b09f3687a26. Latest consensus state: current_term: 1 leader_uuid: "76876e6196f54e7380898b09f3687a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76876e6196f54e7380898b09f3687a26" member_type: VOTER } }
I20240503 08:10:06.111485 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:06.111493 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:06.111730 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:06.111862 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:06.112361 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:06.112442 22407 catalog_manager.cc:1347] Generated new cluster ID: b6185434a98e4796b9f20bf83c791fe4
I20240503 08:10:06.112478 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:06.119833 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:06.120278 22407 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:06.128397 22407 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26: Generated new TSK 0
I20240503 08:10:06.128476 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:06.144270 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:06.146739 22420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:06.147503 22422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:06.148427 22419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:06.149139 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:06.149276 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:06.149324 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:06.149340 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723806149341 us; error 0 us; skew 500 ppm
I20240503 08:10:06.150125 17235 webserver.cc:460] Webserver started at http://127.16.212.193:46069/ using document root <none> and password file <none>
I20240503 08:10:06.150218 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:06.150278 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:06.150336 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:06.150658 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "6f9909972f55474182a6d24e79fdd4f2"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.151628 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:06.152240 22427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.152429 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:06.152485 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "6f9909972f55474182a6d24e79fdd4f2"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.152541 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:06.161455 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:06.161736 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:06.161903 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:06.162115 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:06.162146 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.162185 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:06.162210 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.165223 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:46445
I20240503 08:10:06.166102 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:06.166527 22490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:46445 every 8 connection(s)
W20240503 08:10:06.168054 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:06.168768 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:06.169680 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:06.169919 22491 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39067
I20240503 08:10:06.169978 22491 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:06.170084 22491 heartbeater.cc:511] Master 127.16.212.254:39067 requested a full tablet report, sending...
W20240503 08:10:06.170293 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:06.170368 22362 ts_manager.cc:194] Registered new tserver with Master: 6f9909972f55474182a6d24e79fdd4f2 (127.16.212.193:46445)
I20240503 08:10:06.170475 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:06.170503 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:06.170516 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723806170516 us; error 0 us; skew 500 ppm
I20240503 08:10:06.171175 22362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54132
I20240503 08:10:06.171245 17235 webserver.cc:460] Webserver started at http://127.16.212.194:41931/ using document root <none> and password file <none>
I20240503 08:10:06.171340 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:06.171399 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:06.171450 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:06.171721 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "0e2dab890290459cbf16028352b559c7"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.172555 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:06.172976 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.173112 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.173154 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "0e2dab890290459cbf16028352b559c7"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.173206 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:06.183669 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:06.183867 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:06.184020 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:06.184206 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:06.184233 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.184267 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:06.184288 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.187129 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:34139
I20240503 08:10:06.187179 22566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:34139 every 8 connection(s)
I20240503 08:10:06.187511 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:06.189447 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:06.190284 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:06.190472 22567 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39067
I20240503 08:10:06.190529 22567 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:06.190665 22567 heartbeater.cc:511] Master 127.16.212.254:39067 requested a full tablet report, sending...
I20240503 08:10:06.190980 22362 ts_manager.cc:194] Registered new tserver with Master: 0e2dab890290459cbf16028352b559c7 (127.16.212.194:34139)
W20240503 08:10:06.191061 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:06.191344 22362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54136
I20240503 08:10:06.192368 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:06.192498 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:06.192538 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:06.192556 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723806192556 us; error 0 us; skew 500 ppm
I20240503 08:10:06.193364 17235 webserver.cc:460] Webserver started at http://127.16.212.195:41109/ using document root <none> and password file <none>
I20240503 08:10:06.193460 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:06.193511 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:06.193560 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:06.193899 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "cd22bed67b074f299adbc53d64d0fe47"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.195003 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:06.195502 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.195650 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.195703 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "cd22bed67b074f299adbc53d64d0fe47"
format_stamp: "Formatted at 2024-05-03 08:10:06 on dist-test-slave-zl2t"
I20240503 08:10:06.195768 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:06.232184 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:06.232471 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:06.232712 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:06.232916 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:06.232944 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.232980 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:06.233009 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.236238 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:36585
I20240503 08:10:06.236299 22641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:36585 every 8 connection(s)
I20240503 08:10:06.239909 22642 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39067
I20240503 08:10:06.239987 22642 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:06.240080 22642 heartbeater.cc:511] Master 127.16.212.254:39067 requested a full tablet report, sending...
I20240503 08:10:06.240371 22362 ts_manager.cc:194] Registered new tserver with Master: cd22bed67b074f299adbc53d64d0fe47 (127.16.212.195:36585)
I20240503 08:10:06.240756 22362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54150
I20240503 08:10:06.241009 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004596952s
I20240503 08:10:06.241400 17235 test_util.cc:268] Using random seed: 1062986358
I20240503 08:10:06.249357 22362 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54158:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:06.249536 22362 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:06.255226 22531 tablet_service.cc:1467] Processing CreateTablet for tablet 57e19e4982f345c4815ac0994fe01500 (DEFAULT_TABLE table=test-workload [id=54f267627e064f6f964c3d71e0148a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:06.255398 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e19e4982f345c4815ac0994fe01500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:06.255424 22455 tablet_service.cc:1467] Processing CreateTablet for tablet 57e19e4982f345c4815ac0994fe01500 (DEFAULT_TABLE table=test-workload [id=54f267627e064f6f964c3d71e0148a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:06.255527 22606 tablet_service.cc:1467] Processing CreateTablet for tablet 57e19e4982f345c4815ac0994fe01500 (DEFAULT_TABLE table=test-workload [id=54f267627e064f6f964c3d71e0148a25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:06.255554 22455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e19e4982f345c4815ac0994fe01500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:06.255607 22606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e19e4982f345c4815ac0994fe01500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:06.256659 22657 tablet_bootstrap.cc:492] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7: Bootstrap starting.
I20240503 08:10:06.257035 22657 tablet_bootstrap.cc:654] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:06.257139 22659 tablet_bootstrap.cc:492] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: Bootstrap starting.
I20240503 08:10:06.257364 22658 tablet_bootstrap.cc:492] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47: Bootstrap starting.
I20240503 08:10:06.257683 22657 tablet_bootstrap.cc:492] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7: No bootstrap required, opened a new log
I20240503 08:10:06.257735 22657 ts_tablet_manager.cc:1367] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:06.257817 22658 tablet_bootstrap.cc:654] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:06.257844 22659 tablet_bootstrap.cc:654] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:06.257917 22657 raft_consensus.cc:348] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.257997 22657 raft_consensus.cc:374] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:06.258018 22657 raft_consensus.cc:729] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2dab890290459cbf16028352b559c7, State: Initialized, Role: FOLLOWER
I20240503 08:10:06.258071 22657 consensus_queue.cc:260] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.258227 22657 ts_tablet_manager.cc:1398] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.258273 22567 heartbeater.cc:503] Master 127.16.212.254:39067 was elected leader, sending a full tablet report...
I20240503 08:10:06.258409 22659 tablet_bootstrap.cc:492] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: No bootstrap required, opened a new log
I20240503 08:10:06.258466 22659 ts_tablet_manager.cc:1367] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:06.258484 22658 tablet_bootstrap.cc:492] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47: No bootstrap required, opened a new log
I20240503 08:10:06.258527 22658 ts_tablet_manager.cc:1367] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:06.258605 22659 raft_consensus.cc:348] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.258683 22659 raft_consensus.cc:374] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:06.258725 22659 raft_consensus.cc:729] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9909972f55474182a6d24e79fdd4f2, State: Initialized, Role: FOLLOWER
I20240503 08:10:06.258746 22658 raft_consensus.cc:348] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.258795 22659 consensus_queue.cc:260] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.258834 22658 raft_consensus.cc:374] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:06.258891 22658 raft_consensus.cc:729] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd22bed67b074f299adbc53d64d0fe47, State: Initialized, Role: FOLLOWER
I20240503 08:10:06.258944 22659 ts_tablet_manager.cc:1398] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:06.258970 22491 heartbeater.cc:503] Master 127.16.212.254:39067 was elected leader, sending a full tablet report...
I20240503 08:10:06.259012 22658 consensus_queue.cc:260] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.259212 22658 ts_tablet_manager.cc:1398] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:06.259228 22642 heartbeater.cc:503] Master 127.16.212.254:39067 was elected leader, sending a full tablet report...
W20240503 08:10:06.416496 22492 tablet.cc:2369] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:10:06.661118 22664 raft_consensus.cc:482] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:06.661252 22664 raft_consensus.cc:504] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.661520 22664 leader_election.cc:288] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd22bed67b074f299adbc53d64d0fe47 (127.16.212.195:36585), 0e2dab890290459cbf16028352b559c7 (127.16.212.194:34139)
I20240503 08:10:06.665309 22616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e19e4982f345c4815ac0994fe01500" candidate_uuid: "6f9909972f55474182a6d24e79fdd4f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd22bed67b074f299adbc53d64d0fe47" is_pre_election: true
I20240503 08:10:06.665309 22541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e19e4982f345c4815ac0994fe01500" candidate_uuid: "6f9909972f55474182a6d24e79fdd4f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2dab890290459cbf16028352b559c7" is_pre_election: true
I20240503 08:10:06.665488 22541 raft_consensus.cc:2454] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9909972f55474182a6d24e79fdd4f2 in term 0.
I20240503 08:10:06.665488 22616 raft_consensus.cc:2454] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9909972f55474182a6d24e79fdd4f2 in term 0.
I20240503 08:10:06.665700 22428 leader_election.cc:302] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2dab890290459cbf16028352b559c7, 6f9909972f55474182a6d24e79fdd4f2; no voters:
I20240503 08:10:06.665827 22664 raft_consensus.cc:2781] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:06.665886 22664 raft_consensus.cc:482] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:06.665910 22664 raft_consensus.cc:3037] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:06.666601 22664 raft_consensus.cc:504] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.666769 22664 leader_election.cc:288] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [CANDIDATE]: Term 1 election: Requested vote from peers cd22bed67b074f299adbc53d64d0fe47 (127.16.212.195:36585), 0e2dab890290459cbf16028352b559c7 (127.16.212.194:34139)
I20240503 08:10:06.666903 22616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e19e4982f345c4815ac0994fe01500" candidate_uuid: "6f9909972f55474182a6d24e79fdd4f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd22bed67b074f299adbc53d64d0fe47"
I20240503 08:10:06.666980 22616 raft_consensus.cc:3037] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:06.666954 22541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e19e4982f345c4815ac0994fe01500" candidate_uuid: "6f9909972f55474182a6d24e79fdd4f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2dab890290459cbf16028352b559c7"
I20240503 08:10:06.667019 22541 raft_consensus.cc:3037] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:06.667634 22541 raft_consensus.cc:2454] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9909972f55474182a6d24e79fdd4f2 in term 1.
I20240503 08:10:06.667651 22616 raft_consensus.cc:2454] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9909972f55474182a6d24e79fdd4f2 in term 1.
I20240503 08:10:06.667809 22428 leader_election.cc:302] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2dab890290459cbf16028352b559c7, 6f9909972f55474182a6d24e79fdd4f2; no voters:
I20240503 08:10:06.667896 22664 raft_consensus.cc:2781] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:06.668018 22664 raft_consensus.cc:686] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9909972f55474182a6d24e79fdd4f2, State: Running, Role: LEADER
I20240503 08:10:06.668071 22664 consensus_queue.cc:237] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } }
I20240503 08:10:06.668565 22362 catalog_manager.cc:5484] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9909972f55474182a6d24e79fdd4f2 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "6f9909972f55474182a6d24e79fdd4f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9909972f55474182a6d24e79fdd4f2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 46445 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:06.702437 22541 raft_consensus.cc:1261] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 1 FOLLOWER]: Refusing update from remote peer 6f9909972f55474182a6d24e79fdd4f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:06.702445 22616 raft_consensus.cc:1261] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 1 FOLLOWER]: Refusing update from remote peer 6f9909972f55474182a6d24e79fdd4f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:06.702723 22667 consensus_queue.cc:1035] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e2dab890290459cbf16028352b559c7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:06.702804 22664 consensus_queue.cc:1035] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd22bed67b074f299adbc53d64d0fe47" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:16.710422 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:10:16.713426 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:16.713552 17235 tablet_replica.cc:330] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2: stopping tablet replica
I20240503 08:10:16.713608 17235 raft_consensus.cc:2229] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:16.713660 17235 raft_consensus.cc:2258] T 57e19e4982f345c4815ac0994fe01500 P 6f9909972f55474182a6d24e79fdd4f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:16.731922 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:10:16.732978 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:10:16.735891 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:16.736040 17235 tablet_replica.cc:330] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7: stopping tablet replica
I20240503 08:10:16.736095 17235 raft_consensus.cc:2229] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:16.736148 17235 raft_consensus.cc:2258] T 57e19e4982f345c4815ac0994fe01500 P 0e2dab890290459cbf16028352b559c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:16.749871 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:10:16.750905 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:10:16.754096 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:16.754220 17235 tablet_replica.cc:330] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47: stopping tablet replica
I20240503 08:10:16.754293 17235 raft_consensus.cc:2229] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:16.754343 17235 raft_consensus.cc:2258] T 57e19e4982f345c4815ac0994fe01500 P cd22bed67b074f299adbc53d64d0fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:16.777608 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:10:16.778541 17235 master.cc:537] Master@127.16.212.254:39067 shutting down...
I20240503 08:10:16.780725 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:16.780800 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:16.780820 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 76876e6196f54e7380898b09f3687a26: stopping tablet replica
I20240503 08:10:16.792567 17235 master.cc:552] Master@127.16.212.254:39067 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock (10736 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientCommitSequence
I20240503 08:10:16.811961 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:33819
I20240503 08:10:16.812130 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:16.814163 22716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.814893 22719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.815385 22717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:16.816191 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:16.816296 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:16.816332 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:16.816345 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723816816345 us; error 0 us; skew 500 ppm
I20240503 08:10:16.816965 17235 webserver.cc:460] Webserver started at http://127.16.212.254:33419/ using document root <none> and password file <none>
I20240503 08:10:16.817052 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:16.817104 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:16.817155 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:16.817422 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "7a0b990fa4e44f3dabf8c77bd10a7044"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.818269 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:16.818781 22724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.818944 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.818987 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "7a0b990fa4e44f3dabf8c77bd10a7044"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.819032 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:16.833353 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:16.833547 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:16.836134 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:33819
I20240503 08:10:16.838100 22776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:33819 every 8 connection(s)
I20240503 08:10:16.838202 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:16.839234 22777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044: Bootstrap starting.
I20240503 08:10:16.839466 22777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:16.839969 22777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044: No bootstrap required, opened a new log
I20240503 08:10:16.840065 22777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" member_type: VOTER }
I20240503 08:10:16.840112 22777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:16.840126 22777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0b990fa4e44f3dabf8c77bd10a7044, State: Initialized, Role: FOLLOWER
I20240503 08:10:16.840163 22777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" member_type: VOTER }
I20240503 08:10:16.840205 22777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:16.840227 22777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:16.840248 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:16.840699 22777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" member_type: VOTER }
I20240503 08:10:16.840750 22777 leader_election.cc:302] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a0b990fa4e44f3dabf8c77bd10a7044; no voters:
I20240503 08:10:16.840827 22777 leader_election.cc:288] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:16.840900 22780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:16.840981 22777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:16.841009 22780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0b990fa4e44f3dabf8c77bd10a7044, State: Running, Role: LEADER
I20240503 08:10:16.841051 22780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" member_type: VOTER }
I20240503 08:10:16.841292 22781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" member_type: VOTER } }
I20240503 08:10:16.841387 22781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:16.841702 22785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:16.841847 22785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:16.841998 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a0b990fa4e44f3dabf8c77bd10a7044. Latest consensus state: current_term: 1 leader_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0b990fa4e44f3dabf8c77bd10a7044" member_type: VOTER } }
I20240503 08:10:16.842058 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:16.842388 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:16.842463 22785 catalog_manager.cc:1347] Generated new cluster ID: 1d2e3624bd404cf884497f3556fe2ab6
I20240503 08:10:16.842500 22785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:16.858508 22785 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:16.858960 22785 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:16.866890 22785 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044: Generated new TSK 0
I20240503 08:10:16.866978 22785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:16.874176 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:16.876325 22798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.877104 22799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.877673 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:16.878501 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:16.878603 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:16.878667 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:16.878705 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723816878704 us; error 0 us; skew 500 ppm
I20240503 08:10:16.879387 17235 webserver.cc:460] Webserver started at http://127.16.212.193:34473/ using document root <none> and password file <none>
I20240503 08:10:16.879473 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:16.879521 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:16.879580 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:16.879869 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "26a77e2b986f4a2c8742ef47cba44815"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.880734 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:16.881175 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.881304 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.881346 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "26a77e2b986f4a2c8742ef47cba44815"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.881392 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:16.895704 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:16.895902 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:16.896047 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:16.896240 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:16.896270 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.896303 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:16.896333 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.899156 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:33841
I20240503 08:10:16.899497 22869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:33841 every 8 connection(s)
I20240503 08:10:16.899672 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:16.901531 22874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.902166 22875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.903041 22877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:16.903152 22870 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33819
I20240503 08:10:16.903264 22870 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:16.903378 22870 heartbeater.cc:511] Master 127.16.212.254:33819 requested a full tablet report, sending...
I20240503 08:10:16.903676 22741 ts_manager.cc:194] Registered new tserver with Master: 26a77e2b986f4a2c8742ef47cba44815 (127.16.212.193:33841)
I20240503 08:10:16.903718 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:16.903851 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:16.903883 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:16.903906 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723816903906 us; error 0 us; skew 500 ppm
I20240503 08:10:16.904479 22741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46720
I20240503 08:10:16.904681 17235 webserver.cc:460] Webserver started at http://127.16.212.194:36559/ using document root <none> and password file <none>
I20240503 08:10:16.904774 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:16.904825 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:16.904875 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:16.905133 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "0bba1fb189e34213b3519dc27f169f63"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.906000 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:16.906440 22882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.906579 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.906644 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "0bba1fb189e34213b3519dc27f169f63"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.906695 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:16.922101 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:16.922293 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:16.922456 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:16.922673 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:16.922710 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.922741 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:16.922771 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.925412 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:42001
I20240503 08:10:16.925937 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:16.926379 22945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:42001 every 8 connection(s)
W20240503 08:10:16.927959 22948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.928735 22951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:16.929574 22949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:16.929674 22946 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33819
I20240503 08:10:16.929737 22946 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:16.929834 22946 heartbeater.cc:511] Master 127.16.212.254:33819 requested a full tablet report, sending...
I20240503 08:10:16.930038 22741 ts_manager.cc:194] Registered new tserver with Master: 0bba1fb189e34213b3519dc27f169f63 (127.16.212.194:42001)
I20240503 08:10:16.930346 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:16.930454 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:10:16.930473 22741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46724
W20240503 08:10:16.930522 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:16.930553 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723816930553 us; error 0 us; skew 500 ppm
I20240503 08:10:16.931210 17235 webserver.cc:460] Webserver started at http://127.16.212.195:42469/ using document root <none> and password file <none>
I20240503 08:10:16.931278 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:16.931316 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:16.931345 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:16.931609 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "a2246135d23c4708b559a7982de9d183"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.932454 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:16.932873 22957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.933024 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.933076 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "a2246135d23c4708b559a7982de9d183"
format_stamp: "Formatted at 2024-05-03 08:10:16 on dist-test-slave-zl2t"
I20240503 08:10:16.933116 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:16.958038 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:16.958236 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:16.958412 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:16.958663 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:16.958700 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.958724 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:16.958735 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.961886 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:42541
I20240503 08:10:16.961968 23020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:42541 every 8 connection(s)
I20240503 08:10:16.964982 23021 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33819
I20240503 08:10:16.965035 23021 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:16.965137 23021 heartbeater.cc:511] Master 127.16.212.254:33819 requested a full tablet report, sending...
I20240503 08:10:16.965422 22741 ts_manager.cc:194] Registered new tserver with Master: a2246135d23c4708b559a7982de9d183 (127.16.212.195:42541)
I20240503 08:10:16.965770 22741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46730
I20240503 08:10:16.966454 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004404671s
I20240503 08:10:16.966800 17235 test_util.cc:268] Using random seed: 1073711760
I20240503 08:10:16.971084 22741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46742:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:16.971221 22741 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:16.975498 22910 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6b895ec8404b87ba36c60c354da8b3 (DEFAULT_TABLE table=test-workload [id=ca640c8f10ef4039bd2e175530dc5a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:16.975646 22910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6b895ec8404b87ba36c60c354da8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:16.976363 22834 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6b895ec8404b87ba36c60c354da8b3 (DEFAULT_TABLE table=test-workload [id=ca640c8f10ef4039bd2e175530dc5a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:16.976481 22834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6b895ec8404b87ba36c60c354da8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:16.977077 22985 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6b895ec8404b87ba36c60c354da8b3 (DEFAULT_TABLE table=test-workload [id=ca640c8f10ef4039bd2e175530dc5a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:16.977123 23036 tablet_bootstrap.cc:492] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63: Bootstrap starting.
I20240503 08:10:16.977176 22985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6b895ec8404b87ba36c60c354da8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:16.977499 23036 tablet_bootstrap.cc:654] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:16.977764 23037 tablet_bootstrap.cc:492] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815: Bootstrap starting.
I20240503 08:10:16.978180 23037 tablet_bootstrap.cc:654] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:16.978403 23039 tablet_bootstrap.cc:492] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: Bootstrap starting.
I20240503 08:10:16.978778 23039 tablet_bootstrap.cc:654] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:16.978804 23036 tablet_bootstrap.cc:492] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63: No bootstrap required, opened a new log
I20240503 08:10:16.978850 23036 ts_tablet_manager.cc:1367] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:10:16.978874 23037 tablet_bootstrap.cc:492] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815: No bootstrap required, opened a new log
I20240503 08:10:16.978986 23037 ts_tablet_manager.cc:1367] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:16.979080 23036 raft_consensus.cc:348] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:16.979104 23037 raft_consensus.cc:348] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:16.979161 23037 raft_consensus.cc:374] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:16.979187 23037 raft_consensus.cc:729] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a77e2b986f4a2c8742ef47cba44815, State: Initialized, Role: FOLLOWER
I20240503 08:10:16.979241 23037 consensus_queue.cc:260] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:16.979367 23039 tablet_bootstrap.cc:492] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: No bootstrap required, opened a new log
I20240503 08:10:16.979411 23037 ts_tablet_manager.cc:1398] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:16.979451 22870 heartbeater.cc:503] Master 127.16.212.254:33819 was elected leader, sending a full tablet report...
I20240503 08:10:16.979483 23036 raft_consensus.cc:374] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:16.979507 23036 raft_consensus.cc:729] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bba1fb189e34213b3519dc27f169f63, State: Initialized, Role: FOLLOWER
I20240503 08:10:16.979423 23039 ts_tablet_manager.cc:1367] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:16.979573 23036 consensus_queue.cc:260] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:16.979729 23036 ts_tablet_manager.cc:1398] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:16.979771 22946 heartbeater.cc:503] Master 127.16.212.254:33819 was elected leader, sending a full tablet report...
I20240503 08:10:16.979908 23039 raft_consensus.cc:348] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:16.980072 23039 raft_consensus.cc:374] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:16.980113 23039 raft_consensus.cc:729] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2246135d23c4708b559a7982de9d183, State: Initialized, Role: FOLLOWER
I20240503 08:10:16.980202 23039 consensus_queue.cc:260] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:16.980388 23039 ts_tablet_manager.cc:1398] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:16.980417 23021 heartbeater.cc:503] Master 127.16.212.254:33819 was elected leader, sending a full tablet report...
I20240503 08:10:17.220957 23044 raft_consensus.cc:482] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:17.221091 23044 raft_consensus.cc:504] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:17.221366 23044 leader_election.cc:288] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bba1fb189e34213b3519dc27f169f63 (127.16.212.194:42001), 26a77e2b986f4a2c8742ef47cba44815 (127.16.212.193:33841)
I20240503 08:10:17.223910 22920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3" candidate_uuid: "a2246135d23c4708b559a7982de9d183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bba1fb189e34213b3519dc27f169f63" is_pre_election: true
I20240503 08:10:17.223994 22920 raft_consensus.cc:2454] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2246135d23c4708b559a7982de9d183 in term 0.
I20240503 08:10:17.223970 22844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3" candidate_uuid: "a2246135d23c4708b559a7982de9d183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a77e2b986f4a2c8742ef47cba44815" is_pre_election: true
I20240503 08:10:17.224095 22844 raft_consensus.cc:2454] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2246135d23c4708b559a7982de9d183 in term 0.
I20240503 08:10:17.224287 22960 leader_election.cc:302] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a77e2b986f4a2c8742ef47cba44815, a2246135d23c4708b559a7982de9d183; no voters:
I20240503 08:10:17.224386 23044 raft_consensus.cc:2781] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:17.224433 23044 raft_consensus.cc:482] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:17.224457 23044 raft_consensus.cc:3037] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:17.225085 23044 raft_consensus.cc:504] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:17.225211 23044 leader_election.cc:288] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [CANDIDATE]: Term 1 election: Requested vote from peers 0bba1fb189e34213b3519dc27f169f63 (127.16.212.194:42001), 26a77e2b986f4a2c8742ef47cba44815 (127.16.212.193:33841)
I20240503 08:10:17.225365 22920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3" candidate_uuid: "a2246135d23c4708b559a7982de9d183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bba1fb189e34213b3519dc27f169f63"
I20240503 08:10:17.225390 22844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3" candidate_uuid: "a2246135d23c4708b559a7982de9d183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a77e2b986f4a2c8742ef47cba44815"
I20240503 08:10:17.225435 22844 raft_consensus.cc:3037] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:17.225440 22920 raft_consensus.cc:3037] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:17.226364 22844 raft_consensus.cc:2454] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2246135d23c4708b559a7982de9d183 in term 1.
I20240503 08:10:17.226465 22920 raft_consensus.cc:2454] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2246135d23c4708b559a7982de9d183 in term 1.
I20240503 08:10:17.226555 22960 leader_election.cc:302] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a77e2b986f4a2c8742ef47cba44815, a2246135d23c4708b559a7982de9d183; no voters:
I20240503 08:10:17.226711 23044 raft_consensus.cc:2781] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:17.226871 23044 raft_consensus.cc:686] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 LEADER]: Becoming Leader. State: Replica: a2246135d23c4708b559a7982de9d183, State: Running, Role: LEADER
I20240503 08:10:17.226923 23044 consensus_queue.cc:237] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:17.227504 22741 catalog_manager.cc:5484] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2246135d23c4708b559a7982de9d183 (127.16.212.195). New cstate: current_term: 1 leader_uuid: "a2246135d23c4708b559a7982de9d183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:17.261483 22920 raft_consensus.cc:1261] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 1 FOLLOWER]: Refusing update from remote peer a2246135d23c4708b559a7982de9d183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:17.261503 22844 raft_consensus.cc:1261] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 1 FOLLOWER]: Refusing update from remote peer a2246135d23c4708b559a7982de9d183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:17.261811 23044 consensus_queue.cc:1035] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:17.261924 23046 consensus_queue.cc:1035] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:27.741521 23060 raft_consensus.cc:979] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: : Instructing follower 26a77e2b986f4a2c8742ef47cba44815 to start an election
I20240503 08:10:27.741606 23060 raft_consensus.cc:1067] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 LEADER]: Signalling peer 26a77e2b986f4a2c8742ef47cba44815 to start an election
I20240503 08:10:27.741777 22843 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3"
dest_uuid: "26a77e2b986f4a2c8742ef47cba44815"
from {username='slave'} at 127.0.0.1:56424
I20240503 08:10:27.741856 22843 raft_consensus.cc:482] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:10:27.741891 22843 raft_consensus.cc:3037] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:27.742578 22843 raft_consensus.cc:504] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:27.742784 22843 leader_election.cc:288] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [CANDIDATE]: Term 2 election: Requested vote from peers a2246135d23c4708b559a7982de9d183 (127.16.212.195:42541), 0bba1fb189e34213b3519dc27f169f63 (127.16.212.194:42001)
I20240503 08:10:27.745492 22995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3" candidate_uuid: "26a77e2b986f4a2c8742ef47cba44815" candidate_term: 2 candidate_status { last_received { term: 1 index: 13312 } } ignore_live_leader: true dest_uuid: "a2246135d23c4708b559a7982de9d183"
I20240503 08:10:27.745586 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6b895ec8404b87ba36c60c354da8b3" candidate_uuid: "26a77e2b986f4a2c8742ef47cba44815" candidate_term: 2 candidate_status { last_received { term: 1 index: 13312 } } ignore_live_leader: true dest_uuid: "0bba1fb189e34213b3519dc27f169f63"
I20240503 08:10:27.745591 22995 raft_consensus.cc:3032] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:10:27.745682 22919 raft_consensus.cc:3037] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:27.745687 22995 raft_consensus.cc:729] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2246135d23c4708b559a7982de9d183, State: Running, Role: LEADER
I20240503 08:10:27.745775 22995 consensus_queue.cc:260] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13312, Committed index: 13312, Last appended: 1.13312, Last appended by leader: 13312, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:27.745862 22995 raft_consensus.cc:3037] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:27.746362 22919 raft_consensus.cc:2454] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a77e2b986f4a2c8742ef47cba44815 in term 2.
I20240503 08:10:27.746536 22995 raft_consensus.cc:2454] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a77e2b986f4a2c8742ef47cba44815 in term 2.
I20240503 08:10:27.746546 22807 leader_election.cc:302] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bba1fb189e34213b3519dc27f169f63, 26a77e2b986f4a2c8742ef47cba44815; no voters:
I20240503 08:10:27.746793 23096 raft_consensus.cc:2781] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:27.746917 23096 raft_consensus.cc:686] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 2 LEADER]: Becoming Leader. State: Replica: 26a77e2b986f4a2c8742ef47cba44815, State: Running, Role: LEADER
I20240503 08:10:27.746982 23096 consensus_queue.cc:237] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13312, Committed index: 13312, Last appended: 1.13312, Last appended by leader: 13312, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } }
I20240503 08:10:27.747416 22739 catalog_manager.cc:5484] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 reported cstate change: term changed from 1 to 2, leader changed from a2246135d23c4708b559a7982de9d183 (127.16.212.195) to 26a77e2b986f4a2c8742ef47cba44815 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "26a77e2b986f4a2c8742ef47cba44815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26a77e2b986f4a2c8742ef47cba44815" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33841 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:27.759727 22995 raft_consensus.cc:1261] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 2 FOLLOWER]: Refusing update from remote peer 26a77e2b986f4a2c8742ef47cba44815: Log matching property violated. Preceding OpId in replica: term: 1 index: 13312. Preceding OpId from leader: term: 2 index: 13314. (index mismatch)
I20240503 08:10:27.759727 22919 raft_consensus.cc:1261] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 2 FOLLOWER]: Refusing update from remote peer 26a77e2b986f4a2c8742ef47cba44815: Log matching property violated. Preceding OpId in replica: term: 1 index: 13312. Preceding OpId from leader: term: 2 index: 13314. (index mismatch)
I20240503 08:10:27.759953 23096 consensus_queue.cc:1035] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2246135d23c4708b559a7982de9d183" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 42541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13313, Last known committed idx: 13312, Time since last communication: 0.000s
I20240503 08:10:27.760058 23097 consensus_queue.cc:1035] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bba1fb189e34213b3519dc27f169f63" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13313, Last known committed idx: 13312, Time since last communication: 0.000s
W20240503 08:10:27.761947 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Cannot finalize transaction in state: OPEN
W20240503 08:10:27.762320 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction 0 already open
W20240503 08:10:27.764989 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:10:27.765441 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction 0 commit already in progress
W20240503 08:10:27.768040 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:10:27.768425 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:10:27.768807 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction 0 commit already in progress
W20240503 08:10:27.769168 23109 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47000: Illegal state: Transaction metadata for transaction 0 already set
I20240503 08:10:27.769547 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:10:27.772795 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:27.772956 17235 tablet_replica.cc:330] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815: stopping tablet replica
I20240503 08:10:27.773018 17235 raft_consensus.cc:2229] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:27.773088 17235 raft_consensus.cc:2258] T 0d6b895ec8404b87ba36c60c354da8b3 P 26a77e2b986f4a2c8742ef47cba44815 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:27.787122 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:10:27.788050 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:10:27.791071 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:27.791186 17235 tablet_replica.cc:330] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63: stopping tablet replica
I20240503 08:10:27.791245 17235 raft_consensus.cc:2229] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:27.791299 17235 raft_consensus.cc:2258] T 0d6b895ec8404b87ba36c60c354da8b3 P 0bba1fb189e34213b3519dc27f169f63 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:27.812800 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:10:27.813627 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:10:27.817334 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:27.817436 17235 tablet_replica.cc:330] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183: stopping tablet replica
I20240503 08:10:27.817497 17235 raft_consensus.cc:2229] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:27.817550 17235 raft_consensus.cc:2258] T 0d6b895ec8404b87ba36c60c354da8b3 P a2246135d23c4708b559a7982de9d183 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:27.831230 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:10:27.832391 17235 master.cc:537] Master@127.16.212.254:33819 shutting down...
I20240503 08:10:27.836093 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:27.836185 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:27.836211 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a0b990fa4e44f3dabf8c77bd10a7044: stopping tablet replica
I20240503 08:10:27.848716 17235 master.cc:552] Master@127.16.212.254:33819 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientCommitSequence (11055 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientAbortSequence
I20240503 08:10:27.867291 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:33111
I20240503 08:10:27.867460 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:27.869885 17235 server_base.cc:1034] running on GCE node
W20240503 08:10:27.870594 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.871068 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.871773 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:27.871932 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:27.871966 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:27.871979 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723827871980 us; error 0 us; skew 500 ppm
I20240503 08:10:27.872586 17235 webserver.cc:460] Webserver started at http://127.16.212.254:43183/ using document root <none> and password file <none>
I20240503 08:10:27.872653 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:27.872697 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:27.872733 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:27.872987 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "9e45ed2deccc4fd0a19822a5e7ef7932"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.873833 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:27.874271 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.874405 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.874459 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "9e45ed2deccc4fd0a19822a5e7ef7932"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.874497 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:27.897850 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:27.898028 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:27.900480 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:33111
I20240503 08:10:27.913036 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:33111 every 8 connection(s)
I20240503 08:10:27.913185 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:27.914233 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932: Bootstrap starting.
I20240503 08:10:27.914520 23178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:27.915043 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932: No bootstrap required, opened a new log
I20240503 08:10:27.915163 23178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" member_type: VOTER }
I20240503 08:10:27.915222 23178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:27.915235 23178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e45ed2deccc4fd0a19822a5e7ef7932, State: Initialized, Role: FOLLOWER
I20240503 08:10:27.915290 23178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" member_type: VOTER }
I20240503 08:10:27.915333 23178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:27.915352 23178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:27.915381 23178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:27.915850 23178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" member_type: VOTER }
I20240503 08:10:27.915908 23178 leader_election.cc:302] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e45ed2deccc4fd0a19822a5e7ef7932; no voters:
I20240503 08:10:27.916031 23178 leader_election.cc:288] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:27.916074 23181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:27.916191 23178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:27.916190 23181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 1 LEADER]: Becoming Leader. State: Replica: 9e45ed2deccc4fd0a19822a5e7ef7932, State: Running, Role: LEADER
I20240503 08:10:27.916272 23181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" member_type: VOTER }
I20240503 08:10:27.916471 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" member_type: VOTER } }
I20240503 08:10:27.916481 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e45ed2deccc4fd0a19822a5e7ef7932. Latest consensus state: current_term: 1 leader_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45ed2deccc4fd0a19822a5e7ef7932" member_type: VOTER } }
I20240503 08:10:27.916612 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:27.916644 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:27.916884 23186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:27.917061 23186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:27.917598 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:27.917685 23186 catalog_manager.cc:1347] Generated new cluster ID: 4fb18aeef400423f8f6106f15f849fd9
I20240503 08:10:27.917732 23186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:27.932603 23186 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:27.933104 23186 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:27.938570 23186 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932: Generated new TSK 0
I20240503 08:10:27.938694 23186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:27.949373 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:27.951798 23202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.952581 23200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.953168 23199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:27.954010 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:27.954123 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:27.954157 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:27.954178 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723827954178 us; error 0 us; skew 500 ppm
I20240503 08:10:27.954967 17235 webserver.cc:460] Webserver started at http://127.16.212.193:43369/ using document root <none> and password file <none>
I20240503 08:10:27.955065 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:27.955123 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:27.955183 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:27.955508 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "21a2bde894d04a9d90bcb59822d1e6ce"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.956465 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:27.956936 23207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.957069 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:27.957116 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "21a2bde894d04a9d90bcb59822d1e6ce"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.957168 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:27.961652 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:27.961818 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:27.961952 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:27.962128 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:27.962154 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.962184 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:27.962213 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.964828 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:39441
I20240503 08:10:27.965391 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:27.965716 23270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:39441 every 8 connection(s)
W20240503 08:10:27.967366 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.968102 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.968732 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:27.969242 23271 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33111
I20240503 08:10:27.969293 23271 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:27.969394 23271 heartbeater.cc:511] Master 127.16.212.254:33111 requested a full tablet report, sending...
I20240503 08:10:27.969434 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:27.969733 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:10:27.969754 23142 ts_manager.cc:194] Registered new tserver with Master: 21a2bde894d04a9d90bcb59822d1e6ce (127.16.212.193:39441)
W20240503 08:10:27.969789 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:27.969811 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723827969811 us; error 0 us; skew 500 ppm
I20240503 08:10:27.970315 23142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40910
I20240503 08:10:27.970583 17235 webserver.cc:460] Webserver started at http://127.16.212.194:34409/ using document root <none> and password file <none>
I20240503 08:10:27.970692 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:27.970746 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:27.970800 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:27.971052 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "41cbad46706b4b06862c2cd872d77af8"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.971895 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:27.972353 23283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.972515 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.972563 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "41cbad46706b4b06862c2cd872d77af8"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.972612 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:27.985000 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:27.985263 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:27.985442 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:27.985692 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:27.985725 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.985764 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:27.985796 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.988852 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:36701
I20240503 08:10:27.988898 23346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:36701 every 8 connection(s)
I20240503 08:10:27.989238 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:27.991133 23350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:27.991811 23351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:27.992607 23347 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33111
W20240503 08:10:27.992672 23353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:27.992703 23347 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:27.992834 23347 heartbeater.cc:511] Master 127.16.212.254:33111 requested a full tablet report, sending...
I20240503 08:10:27.993046 23142 ts_manager.cc:194] Registered new tserver with Master: 41cbad46706b4b06862c2cd872d77af8 (127.16.212.194:36701)
I20240503 08:10:27.993242 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:27.993369 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:27.993398 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:27.993410 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723827993411 us; error 0 us; skew 500 ppm
I20240503 08:10:27.993490 23142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40920
I20240503 08:10:27.994179 17235 webserver.cc:460] Webserver started at http://127.16.212.195:39923/ using document root <none> and password file <none>
I20240503 08:10:27.994253 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:27.994305 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:27.994357 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:27.994639 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "06113a0e58d549b3963adce177ccc5d2"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.995530 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:27.995977 23358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.996120 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:27.996165 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "06113a0e58d549b3963adce177ccc5d2"
format_stamp: "Formatted at 2024-05-03 08:10:27 on dist-test-slave-zl2t"
I20240503 08:10:27.996214 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:28.010316 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:28.010706 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:28.010891 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:28.011124 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:28.011158 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:28.011194 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:28.011227 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:28.014550 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:34639
I20240503 08:10:28.014601 23421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:34639 every 8 connection(s)
I20240503 08:10:28.018177 23422 heartbeater.cc:347] Connected to a master server at 127.16.212.254:33111
I20240503 08:10:28.018235 23422 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:28.018321 23422 heartbeater.cc:511] Master 127.16.212.254:33111 requested a full tablet report, sending...
I20240503 08:10:28.018530 23142 ts_manager.cc:194] Registered new tserver with Master: 06113a0e58d549b3963adce177ccc5d2 (127.16.212.195:34639)
I20240503 08:10:28.018862 23142 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40936
I20240503 08:10:28.019222 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00445748s
I20240503 08:10:28.019565 17235 test_util.cc:268] Using random seed: 1084764524
I20240503 08:10:28.024052 23142 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40940:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:28.024181 23142 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:28.028609 23386 tablet_service.cc:1467] Processing CreateTablet for tablet 87fb58895a5742189d6b15e0f2b6157a (DEFAULT_TABLE table=test-workload [id=ffcaf8b9bb404d0184d6e9c4d7e5cf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:28.028781 23386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fb58895a5742189d6b15e0f2b6157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:28.029058 23235 tablet_service.cc:1467] Processing CreateTablet for tablet 87fb58895a5742189d6b15e0f2b6157a (DEFAULT_TABLE table=test-workload [id=ffcaf8b9bb404d0184d6e9c4d7e5cf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:28.029191 23235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fb58895a5742189d6b15e0f2b6157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:28.030066 23311 tablet_service.cc:1467] Processing CreateTablet for tablet 87fb58895a5742189d6b15e0f2b6157a (DEFAULT_TABLE table=test-workload [id=ffcaf8b9bb404d0184d6e9c4d7e5cf14]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:28.030086 23437 tablet_bootstrap.cc:492] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: Bootstrap starting.
I20240503 08:10:28.030175 23311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fb58895a5742189d6b15e0f2b6157a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:28.030480 23438 tablet_bootstrap.cc:492] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce: Bootstrap starting.
I20240503 08:10:28.030721 23437 tablet_bootstrap.cc:654] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:28.030912 23438 tablet_bootstrap.cc:654] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:28.031520 23438 tablet_bootstrap.cc:492] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce: No bootstrap required, opened a new log
I20240503 08:10:28.031561 23437 tablet_bootstrap.cc:492] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: No bootstrap required, opened a new log
I20240503 08:10:28.031584 23438 ts_tablet_manager.cc:1367] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:28.031592 23437 ts_tablet_manager.cc:1367] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 08:10:28.031791 23441 tablet_bootstrap.cc:492] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8: Bootstrap starting.
I20240503 08:10:28.031814 23438 raft_consensus.cc:348] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.031880 23438 raft_consensus.cc:374] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:28.031841 23437 raft_consensus.cc:348] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.031925 23438 raft_consensus.cc:729] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a2bde894d04a9d90bcb59822d1e6ce, State: Initialized, Role: FOLLOWER
I20240503 08:10:28.031936 23437 raft_consensus.cc:374] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:28.031967 23437 raft_consensus.cc:729] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06113a0e58d549b3963adce177ccc5d2, State: Initialized, Role: FOLLOWER
I20240503 08:10:28.031997 23438 consensus_queue.cc:260] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.032016 23437 consensus_queue.cc:260] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.032183 23441 tablet_bootstrap.cc:654] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:28.032202 23437 ts_tablet_manager.cc:1398] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:10:28.032253 23422 heartbeater.cc:503] Master 127.16.212.254:33111 was elected leader, sending a full tablet report...
I20240503 08:10:28.032347 23438 ts_tablet_manager.cc:1398] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:28.032430 23271 heartbeater.cc:503] Master 127.16.212.254:33111 was elected leader, sending a full tablet report...
I20240503 08:10:28.032732 23441 tablet_bootstrap.cc:492] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8: No bootstrap required, opened a new log
I20240503 08:10:28.032788 23441 ts_tablet_manager.cc:1367] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:28.032935 23441 raft_consensus.cc:348] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.032996 23441 raft_consensus.cc:374] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:28.033015 23441 raft_consensus.cc:729] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41cbad46706b4b06862c2cd872d77af8, State: Initialized, Role: FOLLOWER
I20240503 08:10:28.033071 23441 consensus_queue.cc:260] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.033238 23441 ts_tablet_manager.cc:1398] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:28.033265 23347 heartbeater.cc:503] Master 127.16.212.254:33111 was elected leader, sending a full tablet report...
I20240503 08:10:28.131776 23442 raft_consensus.cc:482] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:28.131999 23442 raft_consensus.cc:504] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.132491 23442 leader_election.cc:288] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21a2bde894d04a9d90bcb59822d1e6ce (127.16.212.193:39441), 41cbad46706b4b06862c2cd872d77af8 (127.16.212.194:36701)
I20240503 08:10:28.136211 23245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a" candidate_uuid: "06113a0e58d549b3963adce177ccc5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" is_pre_election: true
I20240503 08:10:28.136229 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a" candidate_uuid: "06113a0e58d549b3963adce177ccc5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41cbad46706b4b06862c2cd872d77af8" is_pre_election: true
I20240503 08:10:28.136406 23245 raft_consensus.cc:2454] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06113a0e58d549b3963adce177ccc5d2 in term 0.
I20240503 08:10:28.136427 23321 raft_consensus.cc:2454] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06113a0e58d549b3963adce177ccc5d2 in term 0.
I20240503 08:10:28.136621 23362 leader_election.cc:302] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06113a0e58d549b3963adce177ccc5d2, 21a2bde894d04a9d90bcb59822d1e6ce; no voters:
I20240503 08:10:28.136770 23442 raft_consensus.cc:2781] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:28.136833 23442 raft_consensus.cc:482] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:28.136860 23442 raft_consensus.cc:3037] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:28.137522 23442 raft_consensus.cc:504] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.137674 23442 leader_election.cc:288] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [CANDIDATE]: Term 1 election: Requested vote from peers 21a2bde894d04a9d90bcb59822d1e6ce (127.16.212.193:39441), 41cbad46706b4b06862c2cd872d77af8 (127.16.212.194:36701)
I20240503 08:10:28.137838 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a" candidate_uuid: "06113a0e58d549b3963adce177ccc5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41cbad46706b4b06862c2cd872d77af8"
I20240503 08:10:28.137902 23321 raft_consensus.cc:3037] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:28.137907 23245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a" candidate_uuid: "06113a0e58d549b3963adce177ccc5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a2bde894d04a9d90bcb59822d1e6ce"
I20240503 08:10:28.138000 23245 raft_consensus.cc:3037] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:28.138681 23245 raft_consensus.cc:2454] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06113a0e58d549b3963adce177ccc5d2 in term 1.
I20240503 08:10:28.138692 23321 raft_consensus.cc:2454] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06113a0e58d549b3963adce177ccc5d2 in term 1.
I20240503 08:10:28.138952 23360 leader_election.cc:302] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06113a0e58d549b3963adce177ccc5d2, 41cbad46706b4b06862c2cd872d77af8; no voters:
I20240503 08:10:28.139066 23442 raft_consensus.cc:2781] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:28.139256 23442 raft_consensus.cc:686] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 06113a0e58d549b3963adce177ccc5d2, State: Running, Role: LEADER
I20240503 08:10:28.139317 23442 consensus_queue.cc:237] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:28.139971 23142 catalog_manager.cc:5484] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06113a0e58d549b3963adce177ccc5d2 (127.16.212.195). New cstate: current_term: 1 leader_uuid: "06113a0e58d549b3963adce177ccc5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } health_report { overall_health: HEALTHY } } }
I20240503 08:10:28.149062 23321 raft_consensus.cc:1261] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 1 FOLLOWER]: Refusing update from remote peer 06113a0e58d549b3963adce177ccc5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:28.149125 23245 raft_consensus.cc:1261] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 1 FOLLOWER]: Refusing update from remote peer 06113a0e58d549b3963adce177ccc5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:28.149374 23447 consensus_queue.cc:1035] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:28.149499 23442 consensus_queue.cc:1035] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:28.151162 23454 mvcc.cc:205] Tried to move back new op lower bound from 7023508800096866304 to 7023508800059146240. Current Snapshot: MvccSnapshot[applied={T|T < 7023508800096866304}]
I20240503 08:10:38.734174 23491 raft_consensus.cc:979] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: : Instructing follower 21a2bde894d04a9d90bcb59822d1e6ce to start an election
I20240503 08:10:38.734274 23491 raft_consensus.cc:1067] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 LEADER]: Signalling peer 21a2bde894d04a9d90bcb59822d1e6ce to start an election
I20240503 08:10:38.734529 23245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a"
dest_uuid: "21a2bde894d04a9d90bcb59822d1e6ce"
from {username='slave'} at 127.0.0.1:54544
I20240503 08:10:38.734643 23245 raft_consensus.cc:482] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:10:38.734684 23245 raft_consensus.cc:3037] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:38.735370 23245 raft_consensus.cc:504] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:38.735548 23245 leader_election.cc:288] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [CANDIDATE]: Term 2 election: Requested vote from peers 41cbad46706b4b06862c2cd872d77af8 (127.16.212.194:36701), 06113a0e58d549b3963adce177ccc5d2 (127.16.212.195:34639)
I20240503 08:10:38.738171 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a" candidate_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 13565 } } ignore_live_leader: true dest_uuid: "06113a0e58d549b3963adce177ccc5d2"
I20240503 08:10:38.738204 23320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fb58895a5742189d6b15e0f2b6157a" candidate_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 13565 } } ignore_live_leader: true dest_uuid: "41cbad46706b4b06862c2cd872d77af8"
I20240503 08:10:38.738263 23396 raft_consensus.cc:3032] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:10:38.738281 23320 raft_consensus.cc:3037] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:38.738296 23396 raft_consensus.cc:729] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 06113a0e58d549b3963adce177ccc5d2, State: Running, Role: LEADER
I20240503 08:10:38.738358 23396 consensus_queue.cc:260] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13565, Committed index: 13565, Last appended: 1.13565, Last appended by leader: 13565, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:38.738451 23396 raft_consensus.cc:3037] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:10:38.739060 23320 raft_consensus.cc:2454] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21a2bde894d04a9d90bcb59822d1e6ce in term 2.
I20240503 08:10:38.739079 23396 raft_consensus.cc:2454] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21a2bde894d04a9d90bcb59822d1e6ce in term 2.
I20240503 08:10:38.739305 23208 leader_election.cc:302] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06113a0e58d549b3963adce177ccc5d2, 21a2bde894d04a9d90bcb59822d1e6ce, 41cbad46706b4b06862c2cd872d77af8; no voters:
I20240503 08:10:38.739537 23499 raft_consensus.cc:2781] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:10:38.739635 23499 raft_consensus.cc:686] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 2 LEADER]: Becoming Leader. State: Replica: 21a2bde894d04a9d90bcb59822d1e6ce, State: Running, Role: LEADER
I20240503 08:10:38.739678 23499 consensus_queue.cc:237] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13565, Committed index: 13565, Last appended: 1.13565, Last appended by leader: 13565, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } }
I20240503 08:10:38.740085 23141 catalog_manager.cc:5484] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce reported cstate change: term changed from 1 to 2, leader changed from 06113a0e58d549b3963adce177ccc5d2 (127.16.212.195) to 21a2bde894d04a9d90bcb59822d1e6ce (127.16.212.193). New cstate: current_term: 2 leader_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21a2bde894d04a9d90bcb59822d1e6ce" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 39441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:38.755147 23396 raft_consensus.cc:1261] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 2 FOLLOWER]: Refusing update from remote peer 21a2bde894d04a9d90bcb59822d1e6ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 13565. Preceding OpId from leader: term: 2 index: 13567. (index mismatch)
I20240503 08:10:38.755153 23320 raft_consensus.cc:1261] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 2 FOLLOWER]: Refusing update from remote peer 21a2bde894d04a9d90bcb59822d1e6ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 13565. Preceding OpId from leader: term: 2 index: 13567. (index mismatch)
I20240503 08:10:38.755388 23500 consensus_queue.cc:1035] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "06113a0e58d549b3963adce177ccc5d2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13566, Last known committed idx: 13565, Time since last communication: 0.000s
I20240503 08:10:38.755488 23499 consensus_queue.cc:1035] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "41cbad46706b4b06862c2cd872d77af8" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 36701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13566, Last known committed idx: 13565, Time since last communication: 0.000s
W20240503 08:10:38.764051 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:10:38.764477 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Transaction metadata for transaction 1 already exists
W20240503 08:10:38.764796 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Cannot begin committing transaction in state: INITIALIZING
W20240503 08:10:38.765080 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Cannot begin committing transaction in state: INITIALIZING
W20240503 08:10:38.765372 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Transaction metadata for transaction 0 already exists
W20240503 08:10:38.765648 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Transaction metadata for transaction 1 already exists
W20240503 08:10:38.765936 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Transaction metadata for transaction 0 already set
W20240503 08:10:38.766227 23513 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39646: Illegal state: Transaction metadata for transaction 1 already set
I20240503 08:10:38.766557 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:10:38.769656 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:38.769795 17235 tablet_replica.cc:330] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce: stopping tablet replica
I20240503 08:10:38.769838 17235 raft_consensus.cc:2229] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:10:38.769891 17235 raft_consensus.cc:2258] T 87fb58895a5742189d6b15e0f2b6157a P 21a2bde894d04a9d90bcb59822d1e6ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:38.784672 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:10:38.785564 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:10:38.788619 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:38.788728 17235 tablet_replica.cc:330] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8: stopping tablet replica
I20240503 08:10:38.788782 17235 raft_consensus.cc:2229] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:38.788831 17235 raft_consensus.cc:2258] T 87fb58895a5742189d6b15e0f2b6157a P 41cbad46706b4b06862c2cd872d77af8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:38.792176 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:10:38.792866 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:10:38.795656 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:38.795764 17235 tablet_replica.cc:330] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2: stopping tablet replica
I20240503 08:10:38.795812 17235 raft_consensus.cc:2229] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:38.795855 17235 raft_consensus.cc:2258] T 87fb58895a5742189d6b15e0f2b6157a P 06113a0e58d549b3963adce177ccc5d2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:38.809377 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:10:38.810154 17235 master.cc:537] Master@127.16.212.254:33111 shutting down...
I20240503 08:10:38.812219 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:38.812291 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:38.812319 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9e45ed2deccc4fd0a19822a5e7ef7932: stopping tablet replica
I20240503 08:10:38.823995 17235 master.cc:552] Master@127.16.212.254:33111 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientAbortSequence (10976 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun
I20240503 08:10:38.843422 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:36525
I20240503 08:10:38.843593 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:38.846266 23524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:38.846750 23522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.847534 17235 server_base.cc:1034] running on GCE node
W20240503 08:10:38.848158 23521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.848285 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:38.848322 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:38.848340 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723838848341 us; error 0 us; skew 500 ppm
I20240503 08:10:38.849124 17235 webserver.cc:460] Webserver started at http://127.16.212.254:40105/ using document root <none> and password file <none>
I20240503 08:10:38.849211 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:38.849251 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:38.849283 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:38.849546 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "a95f0085d8374d31b6999e523d51cd73"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.850441 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:38.850984 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.851127 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.851182 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "a95f0085d8374d31b6999e523d51cd73"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.851233 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:38.864272 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:38.864463 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:38.866963 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:36525
I20240503 08:10:38.873263 23581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:36525 every 8 connection(s)
I20240503 08:10:38.873337 23582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:38.875073 23582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73: Bootstrap starting.
I20240503 08:10:38.875351 23582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:38.875902 23582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73: No bootstrap required, opened a new log
I20240503 08:10:38.876015 23582 raft_consensus.cc:348] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95f0085d8374d31b6999e523d51cd73" member_type: VOTER }
I20240503 08:10:38.876080 23582 raft_consensus.cc:374] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:38.876092 23582 raft_consensus.cc:729] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95f0085d8374d31b6999e523d51cd73, State: Initialized, Role: FOLLOWER
I20240503 08:10:38.876153 23582 consensus_queue.cc:260] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95f0085d8374d31b6999e523d51cd73" member_type: VOTER }
I20240503 08:10:38.876204 23582 raft_consensus.cc:388] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:38.876219 23582 raft_consensus.cc:482] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:38.876233 23582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:38.876673 23582 raft_consensus.cc:504] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95f0085d8374d31b6999e523d51cd73" member_type: VOTER }
I20240503 08:10:38.876730 23582 leader_election.cc:302] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a95f0085d8374d31b6999e523d51cd73; no voters:
I20240503 08:10:38.876830 23582 leader_election.cc:288] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:38.876894 23585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:38.876996 23582 sys_catalog.cc:564] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:38.877007 23585 raft_consensus.cc:686] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 1 LEADER]: Becoming Leader. State: Replica: a95f0085d8374d31b6999e523d51cd73, State: Running, Role: LEADER
I20240503 08:10:38.877164 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95f0085d8374d31b6999e523d51cd73" member_type: VOTER }
I20240503 08:10:38.877418 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a95f0085d8374d31b6999e523d51cd73. Latest consensus state: current_term: 1 leader_uuid: "a95f0085d8374d31b6999e523d51cd73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95f0085d8374d31b6999e523d51cd73" member_type: VOTER } }
I20240503 08:10:38.877418 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a95f0085d8374d31b6999e523d51cd73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a95f0085d8374d31b6999e523d51cd73" member_type: VOTER } }
I20240503 08:10:38.877595 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:38.877534 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:38.877869 23594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:38.878029 23594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:38.878525 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:38.878595 23594 catalog_manager.cc:1347] Generated new cluster ID: d404e5a487ae432cb10997191bfafdb8
I20240503 08:10:38.878661 23594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:38.897368 23594 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:38.897845 23594 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:38.902172 23594 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73: Generated new TSK 0
I20240503 08:10:38.902261 23594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:38.910445 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:38.912889 23604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:38.913609 23606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:38.914270 23603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.915128 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:38.915267 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:38.915303 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:38.915318 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723838915318 us; error 0 us; skew 500 ppm
I20240503 08:10:38.916011 17235 webserver.cc:460] Webserver started at http://127.16.212.193:43427/ using document root <none> and password file <none>
I20240503 08:10:38.916090 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:38.916141 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:38.916175 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:38.916488 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "8847c5040c3c4fa48c9c20988cd0549e"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.917353 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:38.917862 23611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.917986 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.918041 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "8847c5040c3c4fa48c9c20988cd0549e"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.918093 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:38.935061 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:38.935335 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:38.935487 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:38.935681 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:38.935708 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.935741 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:38.935770 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.938447 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:36515
I20240503 08:10:38.939075 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:38.939249 23674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:36515 every 8 connection(s)
W20240503 08:10:38.942785 23677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.942925 23675 heartbeater.cc:347] Connected to a master server at 127.16.212.254:36525
I20240503 08:10:38.942994 23675 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:38.943113 23675 heartbeater.cc:511] Master 127.16.212.254:36525 requested a full tablet report, sending...
I20240503 08:10:38.943382 23546 ts_manager.cc:194] Registered new tserver with Master: 8847c5040c3c4fa48c9c20988cd0549e (127.16.212.193:36515)
W20240503 08:10:38.943455 23678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.943943 23546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38224
I20240503 08:10:38.944465 17235 server_base.cc:1034] running on GCE node
W20240503 08:10:38.945169 23682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.945454 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:38.945502 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:38.945516 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723838945517 us; error 0 us; skew 500 ppm
I20240503 08:10:38.946341 17235 webserver.cc:460] Webserver started at http://127.16.212.194:46165/ using document root <none> and password file <none>
I20240503 08:10:38.946442 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:38.946497 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:38.946545 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:38.946867 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "56e7845e15ae4e17a1f35823cb6698e7"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.947818 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:38.948469 23687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.948827 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:38.948887 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "56e7845e15ae4e17a1f35823cb6698e7"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.948949 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:38.971788 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:38.972028 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:38.972186 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:38.972378 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:38.972404 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.972437 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:38.972463 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.975149 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:33283
I20240503 08:10:38.975230 23750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:33283 every 8 connection(s)
I20240503 08:10:38.975531 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:38.977494 23754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:38.978349 23755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.978895 23751 heartbeater.cc:347] Connected to a master server at 127.16.212.254:36525
W20240503 08:10:38.978950 23757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:38.978956 23751 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:38.979138 23751 heartbeater.cc:511] Master 127.16.212.254:36525 requested a full tablet report, sending...
I20240503 08:10:38.979357 23546 ts_manager.cc:194] Registered new tserver with Master: 56e7845e15ae4e17a1f35823cb6698e7 (127.16.212.194:33283)
I20240503 08:10:38.979717 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:38.979744 23546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38228
I20240503 08:10:38.979833 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:38.979869 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:38.979887 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723838979887 us; error 0 us; skew 500 ppm
I20240503 08:10:38.980564 17235 webserver.cc:460] Webserver started at http://127.16.212.195:38769/ using document root <none> and password file <none>
I20240503 08:10:38.980635 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:38.980665 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:38.980700 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:38.980942 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "efbcf0a4310847c899a5611011e57891"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.981866 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:38.982342 23762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.982461 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:10:38.982503 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "efbcf0a4310847c899a5611011e57891"
format_stamp: "Formatted at 2024-05-03 08:10:38 on dist-test-slave-zl2t"
I20240503 08:10:38.982546 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:38.989807 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:38.989981 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:38.990120 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:38.990293 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:38.990319 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.990339 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:38.990356 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:38.993475 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:46303
I20240503 08:10:38.993541 23825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:46303 every 8 connection(s)
I20240503 08:10:38.996588 23826 heartbeater.cc:347] Connected to a master server at 127.16.212.254:36525
I20240503 08:10:38.996657 23826 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:38.996749 23826 heartbeater.cc:511] Master 127.16.212.254:36525 requested a full tablet report, sending...
I20240503 08:10:38.996999 23546 ts_manager.cc:194] Registered new tserver with Master: efbcf0a4310847c899a5611011e57891 (127.16.212.195:46303)
I20240503 08:10:38.997329 23546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38238
I20240503 08:10:38.998046 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004399754s
I20240503 08:10:38.998374 17235 test_util.cc:268] Using random seed: 1095743334
I20240503 08:10:39.002929 23546 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38246:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:39.003065 23546 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:39.007869 23790 tablet_service.cc:1467] Processing CreateTablet for tablet ad2cc103de0f4cb998834d52b98ddd18 (DEFAULT_TABLE table=test-workload [id=60867abb1ea94ee79fd34d8bf806523c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:39.008026 23790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2cc103de0f4cb998834d52b98ddd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:39.008078 23715 tablet_service.cc:1467] Processing CreateTablet for tablet ad2cc103de0f4cb998834d52b98ddd18 (DEFAULT_TABLE table=test-workload [id=60867abb1ea94ee79fd34d8bf806523c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:39.008199 23715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2cc103de0f4cb998834d52b98ddd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:39.008323 23639 tablet_service.cc:1467] Processing CreateTablet for tablet ad2cc103de0f4cb998834d52b98ddd18 (DEFAULT_TABLE table=test-workload [id=60867abb1ea94ee79fd34d8bf806523c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:39.008432 23639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2cc103de0f4cb998834d52b98ddd18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:39.009652 23841 tablet_bootstrap.cc:492] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891: Bootstrap starting.
I20240503 08:10:39.009826 23842 tablet_bootstrap.cc:492] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e: Bootstrap starting.
I20240503 08:10:39.010141 23841 tablet_bootstrap.cc:654] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:39.010228 23842 tablet_bootstrap.cc:654] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:39.010761 23845 tablet_bootstrap.cc:492] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7: Bootstrap starting.
I20240503 08:10:39.010811 23842 tablet_bootstrap.cc:492] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e: No bootstrap required, opened a new log
I20240503 08:10:39.010864 23842 ts_tablet_manager.cc:1367] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:39.010998 23841 tablet_bootstrap.cc:492] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891: No bootstrap required, opened a new log
I20240503 08:10:39.011044 23841 ts_tablet_manager.cc:1367] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:39.011132 23845 tablet_bootstrap.cc:654] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:39.011204 23841 raft_consensus.cc:348] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.011273 23841 raft_consensus.cc:374] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:39.011288 23841 raft_consensus.cc:729] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbcf0a4310847c899a5611011e57891, State: Initialized, Role: FOLLOWER
I20240503 08:10:39.011327 23841 consensus_queue.cc:260] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.011482 23841 ts_tablet_manager.cc:1398] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:39.011523 23826 heartbeater.cc:503] Master 127.16.212.254:36525 was elected leader, sending a full tablet report...
I20240503 08:10:39.011031 23842 raft_consensus.cc:348] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.011606 23842 raft_consensus.cc:374] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:39.011637 23842 raft_consensus.cc:729] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8847c5040c3c4fa48c9c20988cd0549e, State: Initialized, Role: FOLLOWER
I20240503 08:10:39.011691 23842 consensus_queue.cc:260] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.011727 23845 tablet_bootstrap.cc:492] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7: No bootstrap required, opened a new log
I20240503 08:10:39.011768 23845 ts_tablet_manager.cc:1367] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:39.011859 23842 ts_tablet_manager.cc:1398] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:39.011938 23675 heartbeater.cc:503] Master 127.16.212.254:36525 was elected leader, sending a full tablet report...
I20240503 08:10:39.011906 23845 raft_consensus.cc:348] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.011998 23845 raft_consensus.cc:374] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:39.012153 23845 raft_consensus.cc:729] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e7845e15ae4e17a1f35823cb6698e7, State: Initialized, Role: FOLLOWER
I20240503 08:10:39.012238 23845 consensus_queue.cc:260] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.012408 23845 ts_tablet_manager.cc:1398] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:39.012485 23751 heartbeater.cc:503] Master 127.16.212.254:36525 was elected leader, sending a full tablet report...
I20240503 08:10:39.038187 23848 raft_consensus.cc:482] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:39.038304 23848 raft_consensus.cc:504] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.038511 23848 leader_election.cc:288] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbcf0a4310847c899a5611011e57891 (127.16.212.195:46303), 56e7845e15ae4e17a1f35823cb6698e7 (127.16.212.194:33283)
I20240503 08:10:39.041545 23800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2cc103de0f4cb998834d52b98ddd18" candidate_uuid: "8847c5040c3c4fa48c9c20988cd0549e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbcf0a4310847c899a5611011e57891" is_pre_election: true
I20240503 08:10:39.041570 23725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2cc103de0f4cb998834d52b98ddd18" candidate_uuid: "8847c5040c3c4fa48c9c20988cd0549e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7845e15ae4e17a1f35823cb6698e7" is_pre_election: true
I20240503 08:10:39.041657 23800 raft_consensus.cc:2454] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8847c5040c3c4fa48c9c20988cd0549e in term 0.
I20240503 08:10:39.041683 23725 raft_consensus.cc:2454] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8847c5040c3c4fa48c9c20988cd0549e in term 0.
I20240503 08:10:39.041807 23615 leader_election.cc:302] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8847c5040c3c4fa48c9c20988cd0549e, efbcf0a4310847c899a5611011e57891; no voters:
I20240503 08:10:39.041896 23848 raft_consensus.cc:2781] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:39.041942 23848 raft_consensus.cc:482] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:39.041957 23848 raft_consensus.cc:3037] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:39.042640 23848 raft_consensus.cc:504] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.042773 23848 leader_election.cc:288] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [CANDIDATE]: Term 1 election: Requested vote from peers efbcf0a4310847c899a5611011e57891 (127.16.212.195:46303), 56e7845e15ae4e17a1f35823cb6698e7 (127.16.212.194:33283)
I20240503 08:10:39.042938 23725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2cc103de0f4cb998834d52b98ddd18" candidate_uuid: "8847c5040c3c4fa48c9c20988cd0549e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e7845e15ae4e17a1f35823cb6698e7"
I20240503 08:10:39.043030 23725 raft_consensus.cc:3037] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:39.042918 23800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2cc103de0f4cb998834d52b98ddd18" candidate_uuid: "8847c5040c3c4fa48c9c20988cd0549e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbcf0a4310847c899a5611011e57891"
I20240503 08:10:39.043089 23800 raft_consensus.cc:3037] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:39.043696 23725 raft_consensus.cc:2454] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8847c5040c3c4fa48c9c20988cd0549e in term 1.
I20240503 08:10:39.043725 23800 raft_consensus.cc:2454] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8847c5040c3c4fa48c9c20988cd0549e in term 1.
I20240503 08:10:39.043861 23615 leader_election.cc:302] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8847c5040c3c4fa48c9c20988cd0549e, efbcf0a4310847c899a5611011e57891; no voters:
I20240503 08:10:39.043944 23848 raft_consensus.cc:2781] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:39.044100 23848 raft_consensus.cc:686] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 1 LEADER]: Becoming Leader. State: Replica: 8847c5040c3c4fa48c9c20988cd0549e, State: Running, Role: LEADER
I20240503 08:10:39.044147 23848 consensus_queue.cc:237] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } }
I20240503 08:10:39.044670 23546 catalog_manager.cc:5484] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8847c5040c3c4fa48c9c20988cd0549e (127.16.212.193). New cstate: current_term: 1 leader_uuid: "8847c5040c3c4fa48c9c20988cd0549e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8847c5040c3c4fa48c9c20988cd0549e" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 36515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:39.056732 23725 raft_consensus.cc:1261] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 1 FOLLOWER]: Refusing update from remote peer 8847c5040c3c4fa48c9c20988cd0549e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:39.056756 23800 raft_consensus.cc:1261] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 1 FOLLOWER]: Refusing update from remote peer 8847c5040c3c4fa48c9c20988cd0549e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:39.056990 23851 consensus_queue.cc:1035] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbcf0a4310847c899a5611011e57891" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:39.057152 23848 consensus_queue.cc:1035] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e7845e15ae4e17a1f35823cb6698e7" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 08:10:49.068833 23858 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53528: Illegal state: Transaction hasn't been successfully started
W20240503 08:10:49.069388 23858 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53528: Illegal state: Transaction metadata for transaction 1 does not exist
I20240503 08:10:49.072327 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:10:49.075062 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:49.075168 17235 tablet_replica.cc:330] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e: stopping tablet replica
I20240503 08:10:49.075223 17235 raft_consensus.cc:2229] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:49.075294 17235 raft_consensus.cc:2258] T ad2cc103de0f4cb998834d52b98ddd18 P 8847c5040c3c4fa48c9c20988cd0549e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:49.093271 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:10:49.094138 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:10:49.096606 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:49.096715 17235 tablet_replica.cc:330] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7: stopping tablet replica
I20240503 08:10:49.096766 17235 raft_consensus.cc:2229] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:49.096819 17235 raft_consensus.cc:2258] T ad2cc103de0f4cb998834d52b98ddd18 P 56e7845e15ae4e17a1f35823cb6698e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:49.099958 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:10:49.100606 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:10:49.103415 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:10:49.103539 17235 tablet_replica.cc:330] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891: stopping tablet replica
I20240503 08:10:49.103598 17235 raft_consensus.cc:2229] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 08:10:49.103650 17235 raft_consensus.cc:2258] T ad2cc103de0f4cb998834d52b98ddd18 P efbcf0a4310847c899a5611011e57891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:49.126953 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:10:49.127740 17235 master.cc:537] Master@127.16.212.254:36525 shutting down...
I20240503 08:10:49.129755 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:10:49.129822 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:10:49.129840 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P a95f0085d8374d31b6999e523d51cd73: stopping tablet replica
I20240503 08:10:49.141603 17235 master.cc:552] Master@127.16.212.254:36525 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun (10317 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientRepeatCalls
I20240503 08:10:49.161324 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:40725
I20240503 08:10:49.161474 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:10:49.163853 17235 server_base.cc:1034] running on GCE node
W20240503 08:10:49.164451 23906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:49.165264 23903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:49.165805 23904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:49.165947 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:49.165982 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:49.165994 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723849165995 us; error 0 us; skew 500 ppm
I20240503 08:10:49.166679 17235 webserver.cc:460] Webserver started at http://127.16.212.254:45969/ using document root <none> and password file <none>
I20240503 08:10:49.166749 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:49.166781 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:49.166816 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:49.167053 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "8b0347724c37435fbb1f7d93c1333e4c"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.167881 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:49.168288 23911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.168416 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.168462 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "8b0347724c37435fbb1f7d93c1333e4c"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.168514 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:49.180186 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:49.180371 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:49.183064 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:40725
I20240503 08:10:49.185472 23963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:40725 every 8 connection(s)
I20240503 08:10:49.185592 23964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:49.186553 23964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c: Bootstrap starting.
I20240503 08:10:49.186859 23964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:49.187419 23964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c: No bootstrap required, opened a new log
I20240503 08:10:49.187527 23964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0347724c37435fbb1f7d93c1333e4c" member_type: VOTER }
I20240503 08:10:49.187590 23964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:49.187604 23964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0347724c37435fbb1f7d93c1333e4c, State: Initialized, Role: FOLLOWER
I20240503 08:10:49.187655 23964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0347724c37435fbb1f7d93c1333e4c" member_type: VOTER }
I20240503 08:10:49.187705 23964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:10:49.187726 23964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:10:49.187748 23964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:49.188247 23964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0347724c37435fbb1f7d93c1333e4c" member_type: VOTER }
I20240503 08:10:49.188304 23964 leader_election.cc:302] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b0347724c37435fbb1f7d93c1333e4c; no voters:
I20240503 08:10:49.188406 23964 leader_election.cc:288] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:10:49.188480 23967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:49.188540 23964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:10:49.188634 23967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 1 LEADER]: Becoming Leader. State: Replica: 8b0347724c37435fbb1f7d93c1333e4c, State: Running, Role: LEADER
I20240503 08:10:49.188691 23967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0347724c37435fbb1f7d93c1333e4c" member_type: VOTER }
I20240503 08:10:49.188901 23968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b0347724c37435fbb1f7d93c1333e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0347724c37435fbb1f7d93c1333e4c" member_type: VOTER } }
I20240503 08:10:49.188915 23969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b0347724c37435fbb1f7d93c1333e4c. Latest consensus state: current_term: 1 leader_uuid: "8b0347724c37435fbb1f7d93c1333e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0347724c37435fbb1f7d93c1333e4c" member_type: VOTER } }
I20240503 08:10:49.188982 23968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:49.188989 23969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [sys.catalog]: This master's current role is: LEADER
I20240503 08:10:49.189252 23974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:10:49.189405 23974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:10:49.189770 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:10:49.190204 23974 catalog_manager.cc:1347] Generated new cluster ID: 24b60cf7befb4420b920221dc31ee289
I20240503 08:10:49.190250 23974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:10:49.200297 23974 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:10:49.200690 23974 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:10:49.211136 23974 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c: Generated new TSK 0
I20240503 08:10:49.211239 23974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:10:49.221635 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:49.223594 23985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:49.224476 23986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:49.225256 23988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:49.225976 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:49.226073 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:49.226114 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:49.226137 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723849226137 us; error 0 us; skew 500 ppm
I20240503 08:10:49.226830 17235 webserver.cc:460] Webserver started at http://127.16.212.193:35061/ using document root <none> and password file <none>
I20240503 08:10:49.226912 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:49.226974 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:49.227027 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:49.227294 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "0b9f133f89a04ab0aa5fcca71b38ea06"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.228178 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:49.228598 23993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.228724 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.228765 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "0b9f133f89a04ab0aa5fcca71b38ea06"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.228816 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:49.235635 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:49.235812 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:49.235945 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:49.236140 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:49.236166 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.236198 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:49.236227 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.239004 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:43007
I20240503 08:10:49.239245 24056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:43007 every 8 connection(s)
I20240503 08:10:49.239544 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:49.241881 24061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:49.242549 24062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:49.243235 24057 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40725
I20240503 08:10:49.243310 24057 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:49.243429 24057 heartbeater.cc:511] Master 127.16.212.254:40725 requested a full tablet report, sending...
W20240503 08:10:49.243511 24064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:49.243672 23928 ts_manager.cc:194] Registered new tserver with Master: 0b9f133f89a04ab0aa5fcca71b38ea06 (127.16.212.193:43007)
I20240503 08:10:49.244184 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:49.244253 23928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50064
I20240503 08:10:49.244297 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:49.244330 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:49.244360 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723849244360 us; error 0 us; skew 500 ppm
I20240503 08:10:49.245134 17235 webserver.cc:460] Webserver started at http://127.16.212.194:41517/ using document root <none> and password file <none>
I20240503 08:10:49.245204 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:49.245236 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:49.245272 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:49.245519 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "2efd7587805e4bac8d045f224cb920ea"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.246901 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:49.247972 24069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.248271 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:10:49.248319 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "2efd7587805e4bac8d045f224cb920ea"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.248355 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:49.254413 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:49.254638 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:49.254782 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:49.254967 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:49.254997 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.255028 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:49.255057 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.257668 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:43445
I20240503 08:10:49.257721 24132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:43445 every 8 connection(s)
I20240503 08:10:49.258028 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:10:49.259989 24137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:10:49.260673 24136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:49.261197 24133 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40725
I20240503 08:10:49.261293 24133 heartbeater.cc:464] Registering TS with master...
W20240503 08:10:49.261449 24139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:10:49.261476 24133 heartbeater.cc:511] Master 127.16.212.254:40725 requested a full tablet report, sending...
I20240503 08:10:49.261718 23928 ts_manager.cc:194] Registered new tserver with Master: 2efd7587805e4bac8d045f224cb920ea (127.16.212.194:43445)
I20240503 08:10:49.262068 17235 server_base.cc:1034] running on GCE node
I20240503 08:10:49.262163 23928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50080
I20240503 08:10:49.262163 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:10:49.262248 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:10:49.262296 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723849262296 us; error 0 us; skew 500 ppm
I20240503 08:10:49.263190 17235 webserver.cc:460] Webserver started at http://127.16.212.195:39265/ using document root <none> and password file <none>
I20240503 08:10:49.263268 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:10:49.263299 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:10:49.263335 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:10:49.263593 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "c9b6aeea8a144bb6891f16b4399c7647"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.264392 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:49.264808 24144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.264936 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.264971 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "c9b6aeea8a144bb6891f16b4399c7647"
format_stamp: "Formatted at 2024-05-03 08:10:49 on dist-test-slave-zl2t"
I20240503 08:10:49.264997 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:10:49.282469 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:10:49.282733 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:10:49.282881 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:10:49.283046 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:10:49.283068 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.283095 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:10:49.283119 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.286281 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:43061
I20240503 08:10:49.286355 24207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:43061 every 8 connection(s)
I20240503 08:10:49.289558 24208 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40725
I20240503 08:10:49.289620 24208 heartbeater.cc:464] Registering TS with master...
I20240503 08:10:49.289703 24208 heartbeater.cc:511] Master 127.16.212.254:40725 requested a full tablet report, sending...
I20240503 08:10:49.289999 23928 ts_manager.cc:194] Registered new tserver with Master: c9b6aeea8a144bb6891f16b4399c7647 (127.16.212.195:43061)
I20240503 08:10:49.290355 23928 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50082
I20240503 08:10:49.290925 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004469094s
I20240503 08:10:49.291263 17235 test_util.cc:268] Using random seed: 1106036222
I20240503 08:10:49.295874 23928 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50086:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:10:49.296005 23928 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:10:49.300407 24097 tablet_service.cc:1467] Processing CreateTablet for tablet 7db1dfc29197490a833449926b3e78cc (DEFAULT_TABLE table=test-workload [id=5cf5fdf4b93943bf98ecb55daa5ef23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:49.300577 24097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db1dfc29197490a833449926b3e78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:49.301959 24223 tablet_bootstrap.cc:492] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: Bootstrap starting.
I20240503 08:10:49.302152 24172 tablet_service.cc:1467] Processing CreateTablet for tablet 7db1dfc29197490a833449926b3e78cc (DEFAULT_TABLE table=test-workload [id=5cf5fdf4b93943bf98ecb55daa5ef23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:49.302213 24021 tablet_service.cc:1467] Processing CreateTablet for tablet 7db1dfc29197490a833449926b3e78cc (DEFAULT_TABLE table=test-workload [id=5cf5fdf4b93943bf98ecb55daa5ef23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:10:49.302265 24172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db1dfc29197490a833449926b3e78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:49.302311 24021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db1dfc29197490a833449926b3e78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:10:49.302388 24223 tablet_bootstrap.cc:654] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:49.302990 24223 tablet_bootstrap.cc:492] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: No bootstrap required, opened a new log
I20240503 08:10:49.303050 24223 ts_tablet_manager.cc:1367] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:49.303190 24223 raft_consensus.cc:348] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.303259 24223 raft_consensus.cc:374] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:49.303279 24223 raft_consensus.cc:729] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efd7587805e4bac8d045f224cb920ea, State: Initialized, Role: FOLLOWER
I20240503 08:10:49.303320 24223 consensus_queue.cc:260] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.303511 24223 ts_tablet_manager.cc:1398] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:49.303546 24133 heartbeater.cc:503] Master 127.16.212.254:40725 was elected leader, sending a full tablet report...
I20240503 08:10:49.303817 24226 tablet_bootstrap.cc:492] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647: Bootstrap starting.
I20240503 08:10:49.303941 24227 tablet_bootstrap.cc:492] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: Bootstrap starting.
I20240503 08:10:49.304319 24227 tablet_bootstrap.cc:654] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:49.304407 24226 tablet_bootstrap.cc:654] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647: Neither blocks nor log segments found. Creating new log.
I20240503 08:10:49.304932 24227 tablet_bootstrap.cc:492] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: No bootstrap required, opened a new log
I20240503 08:10:49.304967 24226 tablet_bootstrap.cc:492] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647: No bootstrap required, opened a new log
I20240503 08:10:49.304980 24227 ts_tablet_manager.cc:1367] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:49.305042 24226 ts_tablet_manager.cc:1367] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:10:49.305162 24227 raft_consensus.cc:348] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.305181 24226 raft_consensus.cc:348] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.305214 24227 raft_consensus.cc:374] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:49.305228 24226 raft_consensus.cc:374] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:10:49.305240 24227 raft_consensus.cc:729] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9f133f89a04ab0aa5fcca71b38ea06, State: Initialized, Role: FOLLOWER
I20240503 08:10:49.305244 24226 raft_consensus.cc:729] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b6aeea8a144bb6891f16b4399c7647, State: Initialized, Role: FOLLOWER
I20240503 08:10:49.305312 24227 consensus_queue.cc:260] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.305341 24226 consensus_queue.cc:260] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.305466 24227 ts_tablet_manager.cc:1398] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:10:49.305490 24226 ts_tablet_manager.cc:1398] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:10:49.305516 24057 heartbeater.cc:503] Master 127.16.212.254:40725 was elected leader, sending a full tablet report...
I20240503 08:10:49.305546 24208 heartbeater.cc:503] Master 127.16.212.254:40725 was elected leader, sending a full tablet report...
W20240503 08:10:49.489545 24058 tablet.cc:2369] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:10:49.661971 24225 raft_consensus.cc:482] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:10:49.662086 24225 raft_consensus.cc:504] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.662249 24225 leader_election.cc:288] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9b6aeea8a144bb6891f16b4399c7647 (127.16.212.195:43061), 0b9f133f89a04ab0aa5fcca71b38ea06 (127.16.212.193:43007)
I20240503 08:10:49.664894 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc" candidate_uuid: "2efd7587805e4bac8d045f224cb920ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b6aeea8a144bb6891f16b4399c7647" is_pre_election: true
I20240503 08:10:49.664969 24031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc" candidate_uuid: "2efd7587805e4bac8d045f224cb920ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" is_pre_election: true
I20240503 08:10:49.665005 24182 raft_consensus.cc:2454] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efd7587805e4bac8d045f224cb920ea in term 0.
I20240503 08:10:49.665036 24031 raft_consensus.cc:2454] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efd7587805e4bac8d045f224cb920ea in term 0.
I20240503 08:10:49.665179 24071 leader_election.cc:302] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efd7587805e4bac8d045f224cb920ea, c9b6aeea8a144bb6891f16b4399c7647; no voters:
I20240503 08:10:49.665272 24225 raft_consensus.cc:2781] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:10:49.665321 24225 raft_consensus.cc:482] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:10:49.665342 24225 raft_consensus.cc:3037] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:49.666024 24225 raft_consensus.cc:504] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.666131 24225 leader_election.cc:288] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [CANDIDATE]: Term 1 election: Requested vote from peers c9b6aeea8a144bb6891f16b4399c7647 (127.16.212.195:43061), 0b9f133f89a04ab0aa5fcca71b38ea06 (127.16.212.193:43007)
I20240503 08:10:49.666342 24031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc" candidate_uuid: "2efd7587805e4bac8d045f224cb920ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06"
I20240503 08:10:49.666391 24031 raft_consensus.cc:3037] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:49.666378 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc" candidate_uuid: "2efd7587805e4bac8d045f224cb920ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b6aeea8a144bb6891f16b4399c7647"
I20240503 08:10:49.666464 24182 raft_consensus.cc:3037] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:10:49.667080 24031 raft_consensus.cc:2454] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efd7587805e4bac8d045f224cb920ea in term 1.
I20240503 08:10:49.667161 24182 raft_consensus.cc:2454] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efd7587805e4bac8d045f224cb920ea in term 1.
I20240503 08:10:49.667249 24071 leader_election.cc:302] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9f133f89a04ab0aa5fcca71b38ea06, 2efd7587805e4bac8d045f224cb920ea; no voters:
I20240503 08:10:49.667331 24225 raft_consensus.cc:2781] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:10:49.667466 24225 raft_consensus.cc:686] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 LEADER]: Becoming Leader. State: Replica: 2efd7587805e4bac8d045f224cb920ea, State: Running, Role: LEADER
I20240503 08:10:49.667517 24225 consensus_queue.cc:237] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:10:49.668183 23927 catalog_manager.cc:5484] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efd7587805e4bac8d045f224cb920ea (127.16.212.194). New cstate: current_term: 1 leader_uuid: "2efd7587805e4bac8d045f224cb920ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } health_report { overall_health: UNKNOWN } } }
I20240503 08:10:49.751274 24031 raft_consensus.cc:1261] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 1 FOLLOWER]: Refusing update from remote peer 2efd7587805e4bac8d045f224cb920ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:49.751273 24182 raft_consensus.cc:1261] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 1 FOLLOWER]: Refusing update from remote peer 2efd7587805e4bac8d045f224cb920ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:10:49.751519 24233 consensus_queue.cc:1035] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:10:49.751659 24225 consensus_queue.cc:1035] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:00.229369 24258 raft_consensus.cc:979] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: : Instructing follower 0b9f133f89a04ab0aa5fcca71b38ea06 to start an election
I20240503 08:11:00.229471 24258 raft_consensus.cc:1067] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 LEADER]: Signalling peer 0b9f133f89a04ab0aa5fcca71b38ea06 to start an election
I20240503 08:11:00.229763 24030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc"
dest_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06"
from {username='slave'} at 127.0.0.1:54114
I20240503 08:11:00.229858 24030 raft_consensus.cc:482] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:00.229880 24030 raft_consensus.cc:3037] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:00.230561 24030 raft_consensus.cc:504] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:11:00.230762 24030 leader_election.cc:288] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [CANDIDATE]: Term 2 election: Requested vote from peers 2efd7587805e4bac8d045f224cb920ea (127.16.212.194:43445), c9b6aeea8a144bb6891f16b4399c7647 (127.16.212.195:43061)
I20240503 08:11:00.233461 24182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc" candidate_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" candidate_term: 2 candidate_status { last_received { term: 1 index: 13520 } } ignore_live_leader: true dest_uuid: "c9b6aeea8a144bb6891f16b4399c7647"
I20240503 08:11:00.233480 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db1dfc29197490a833449926b3e78cc" candidate_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" candidate_term: 2 candidate_status { last_received { term: 1 index: 13520 } } ignore_live_leader: true dest_uuid: "2efd7587805e4bac8d045f224cb920ea"
I20240503 08:11:00.233546 24182 raft_consensus.cc:3037] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:00.233552 24107 raft_consensus.cc:3032] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:00.233603 24107 raft_consensus.cc:729] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efd7587805e4bac8d045f224cb920ea, State: Running, Role: LEADER
I20240503 08:11:00.233670 24107 consensus_queue.cc:260] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13520, Committed index: 13520, Last appended: 1.13520, Last appended by leader: 13520, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:11:00.233757 24107 raft_consensus.cc:3037] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:00.234357 24107 raft_consensus.cc:2454] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b9f133f89a04ab0aa5fcca71b38ea06 in term 2.
I20240503 08:11:00.234365 24182 raft_consensus.cc:2454] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b9f133f89a04ab0aa5fcca71b38ea06 in term 2.
I20240503 08:11:00.234575 23997 leader_election.cc:302] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9f133f89a04ab0aa5fcca71b38ea06, 2efd7587805e4bac8d045f224cb920ea; no voters:
I20240503 08:11:00.234771 24278 raft_consensus.cc:2781] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:00.234932 24278 raft_consensus.cc:686] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 2 LEADER]: Becoming Leader. State: Replica: 0b9f133f89a04ab0aa5fcca71b38ea06, State: Running, Role: LEADER
I20240503 08:11:00.234987 24278 consensus_queue.cc:237] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13520, Committed index: 13520, Last appended: 1.13520, Last appended by leader: 13520, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } }
I20240503 08:11:00.235482 23926 catalog_manager.cc:5484] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 reported cstate change: term changed from 1 to 2, leader changed from 2efd7587805e4bac8d045f224cb920ea (127.16.212.194) to 0b9f133f89a04ab0aa5fcca71b38ea06 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b9f133f89a04ab0aa5fcca71b38ea06" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 43007 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:00.247442 24107 raft_consensus.cc:1261] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 2 FOLLOWER]: Refusing update from remote peer 0b9f133f89a04ab0aa5fcca71b38ea06: Log matching property violated. Preceding OpId in replica: term: 1 index: 13520. Preceding OpId from leader: term: 2 index: 13522. (index mismatch)
I20240503 08:11:00.247499 24182 raft_consensus.cc:1261] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 2 FOLLOWER]: Refusing update from remote peer 0b9f133f89a04ab0aa5fcca71b38ea06: Log matching property violated. Preceding OpId in replica: term: 1 index: 13520. Preceding OpId from leader: term: 2 index: 13522. (index mismatch)
I20240503 08:11:00.247709 24279 consensus_queue.cc:1035] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efd7587805e4bac8d045f224cb920ea" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13521, Last known committed idx: 13520, Time since last communication: 0.000s
I20240503 08:11:00.247807 24278 consensus_queue.cc:1035] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9b6aeea8a144bb6891f16b4399c7647" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13521, Last known committed idx: 13520, Time since last communication: 0.000s
I20240503 08:11:00.249087 24293 mvcc.cc:205] Tried to move back new op lower bound from 7023508931572191232 to 7023508931523170304. Current Snapshot: MvccSnapshot[applied={T|T < 7023508931572191232}]
W20240503 08:11:00.249608 24291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Illegal state: Transaction 1 already open
W20240503 08:11:00.251026 24291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Illegal state: Transaction 1 commit already in progress
W20240503 08:11:00.252403 24291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Illegal state: Transaction metadata for transaction 1 already set
W20240503 08:11:00.253638 24291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Illegal state: Transaction 2 already open
W20240503 08:11:00.254804 24291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Illegal state: Transaction 2 commit already in progress
W20240503 08:11:00.255996 24291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Illegal state: Transaction metadata for transaction 2 already set
I20240503 08:11:00.256400 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:11:00.258967 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:00.259068 17235 tablet_replica.cc:330] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06: stopping tablet replica
I20240503 08:11:00.259124 17235 raft_consensus.cc:2229] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:11:00.259179 17235 raft_consensus.cc:2258] T 7db1dfc29197490a833449926b3e78cc P 0b9f133f89a04ab0aa5fcca71b38ea06 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:00.272915 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:11:00.273710 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:11:00.276372 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:00.276473 17235 tablet_replica.cc:330] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea: stopping tablet replica
I20240503 08:11:00.276525 17235 raft_consensus.cc:2229] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:00.276579 17235 raft_consensus.cc:2258] T 7db1dfc29197490a833449926b3e78cc P 2efd7587805e4bac8d045f224cb920ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:00.297132 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:11:00.298014 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:11:00.300958 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:00.301087 17235 tablet_replica.cc:330] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647: stopping tablet replica
I20240503 08:11:00.301131 17235 raft_consensus.cc:2229] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:00.301169 17235 raft_consensus.cc:2258] T 7db1dfc29197490a833449926b3e78cc P c9b6aeea8a144bb6891f16b4399c7647 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:00.314533 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:11:00.315452 17235 master.cc:537] Master@127.16.212.254:40725 shutting down...
I20240503 08:11:00.317616 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:00.317693 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:00.317728 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b0347724c37435fbb1f7d93c1333e4c: stopping tablet replica
I20240503 08:11:00.329530 17235 master.cc:552] Master@127.16.212.254:40725 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientRepeatCalls (11186 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority
I20240503 08:11:00.347882 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:43563
I20240503 08:11:00.348054 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:00.350878 17235 server_base.cc:1034] running on GCE node
W20240503 08:11:00.351802 24301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:00.352427 24300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:00.352958 24303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.353118 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:00.353158 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:00.353178 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723860353178 us; error 0 us; skew 500 ppm
I20240503 08:11:00.353852 17235 webserver.cc:460] Webserver started at http://127.16.212.254:46561/ using document root <none> and password file <none>
I20240503 08:11:00.353920 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:00.353971 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:00.354025 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:00.354327 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "1d27af3660fd4a7b8e6292653fd1e03e"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.355281 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:00.355728 24308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.355876 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.355932 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "1d27af3660fd4a7b8e6292653fd1e03e"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.355973 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:00.390977 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:00.391168 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:00.393632 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:43563
I20240503 08:11:00.403497 24361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:00.405206 24360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:43563 every 8 connection(s)
I20240503 08:11:00.405864 24361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e: Bootstrap starting.
I20240503 08:11:00.406117 24361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:00.406599 24361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e: No bootstrap required, opened a new log
I20240503 08:11:00.406713 24361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" member_type: VOTER }
I20240503 08:11:00.406762 24361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:00.406776 24361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d27af3660fd4a7b8e6292653fd1e03e, State: Initialized, Role: FOLLOWER
I20240503 08:11:00.406812 24361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" member_type: VOTER }
I20240503 08:11:00.406854 24361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:00.406878 24361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:00.406906 24361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:00.407351 24361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" member_type: VOTER }
I20240503 08:11:00.407400 24361 leader_election.cc:302] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d27af3660fd4a7b8e6292653fd1e03e; no voters:
I20240503 08:11:00.407483 24361 leader_election.cc:288] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:00.407564 24364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:00.407639 24361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:00.407703 24364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 1 LEADER]: Becoming Leader. State: Replica: 1d27af3660fd4a7b8e6292653fd1e03e, State: Running, Role: LEADER
I20240503 08:11:00.407753 24364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" member_type: VOTER }
I20240503 08:11:00.407967 24366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d27af3660fd4a7b8e6292653fd1e03e. Latest consensus state: current_term: 1 leader_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" member_type: VOTER } }
I20240503 08:11:00.408105 24366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:00.407968 24365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d27af3660fd4a7b8e6292653fd1e03e" member_type: VOTER } }
I20240503 08:11:00.408319 24365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:00.408361 24371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:00.408548 24371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:00.409077 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:00.409106 24371 catalog_manager.cc:1347] Generated new cluster ID: add0404937b145ce8d1399bf1acaaace
I20240503 08:11:00.409135 24371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:00.424429 24371 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:00.424831 24371 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:00.430204 24371 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e: Generated new TSK 0
I20240503 08:11:00.430269 24371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:00.440898 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:00.443509 24385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:00.444227 24382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.444875 17235 server_base.cc:1034] running on GCE node
W20240503 08:11:00.445746 24383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.445933 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:00.445986 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:00.446013 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723860446012 us; error 0 us; skew 500 ppm
I20240503 08:11:00.446801 17235 webserver.cc:460] Webserver started at http://127.16.212.193:42769/ using document root <none> and password file <none>
I20240503 08:11:00.446882 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:00.446952 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:00.447010 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:00.447297 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.448372 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:00.448957 24390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.449108 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:00.449177 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.449229 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:00.475205 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:00.475418 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:00.475586 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:00.475790 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:00.475818 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.475853 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:00.475881 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.478515 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:41797
I20240503 08:11:00.478580 24453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:41797 every 8 connection(s)
I20240503 08:11:00.478919 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:00.481076 24458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:00.481971 24459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.482304 24454 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43563
I20240503 08:11:00.482360 24454 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:00.482440 24454 heartbeater.cc:511] Master 127.16.212.254:43563 requested a full tablet report, sending...
W20240503 08:11:00.482678 24461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.482780 24325 ts_manager.cc:194] Registered new tserver with Master: 6bbcd4f100ef4d95a1f1de6bb4849d81 (127.16.212.193:41797)
I20240503 08:11:00.483325 24325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37496
I20240503 08:11:00.483409 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:00.483608 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:00.483651 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:00.483686 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723860483687 us; error 0 us; skew 500 ppm
I20240503 08:11:00.484422 17235 webserver.cc:460] Webserver started at http://127.16.212.194:37981/ using document root <none> and password file <none>
I20240503 08:11:00.484506 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:00.484567 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:00.484618 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:00.484890 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "ca7e68b53494449eaa02c41000022543"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.485803 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:00.486313 24466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.486449 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.486495 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "ca7e68b53494449eaa02c41000022543"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.486541 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:00.498432 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:00.498690 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:00.498836 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:00.499032 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:00.499059 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.499094 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:00.499121 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.501865 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:34217
I20240503 08:11:00.501926 24529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:34217 every 8 connection(s)
I20240503 08:11:00.502230 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:00.504302 24533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:00.504884 24534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.505162 24530 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43563
I20240503 08:11:00.505216 24530 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:00.505311 24530 heartbeater.cc:511] Master 127.16.212.254:43563 requested a full tablet report, sending...
W20240503 08:11:00.505504 24536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:00.505540 24325 ts_manager.cc:194] Registered new tserver with Master: ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217)
I20240503 08:11:00.505908 24325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37510
I20240503 08:11:00.506433 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:00.506587 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:00.506659 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:00.506692 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723860506691 us; error 0 us; skew 500 ppm
I20240503 08:11:00.507642 17235 webserver.cc:460] Webserver started at http://127.16.212.195:37901/ using document root <none> and password file <none>
I20240503 08:11:00.507725 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:00.507772 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:00.507819 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:00.508131 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "e9768e075afb4a2e962ab69c85fc526a"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.509065 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:00.509562 24541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.509722 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:00.509778 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "e9768e075afb4a2e962ab69c85fc526a"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:00.509830 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:00.520738 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:00.520972 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:00.521159 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:00.521347 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:00.521374 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.521409 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:00.521435 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.524669 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:39581
I20240503 08:11:00.524755 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:39581 every 8 connection(s)
I20240503 08:11:00.527894 24605 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43563
I20240503 08:11:00.527954 24605 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:00.528048 24605 heartbeater.cc:511] Master 127.16.212.254:43563 requested a full tablet report, sending...
I20240503 08:11:00.528280 24325 ts_manager.cc:194] Registered new tserver with Master: e9768e075afb4a2e962ab69c85fc526a (127.16.212.195:39581)
I20240503 08:11:00.528365 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003330615s
I20240503 08:11:00.528754 24325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37522
I20240503 08:11:00.528764 17235 test_util.cc:268] Using random seed: 1117273722
I20240503 08:11:00.533296 24325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37538:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:11:00.533428 24325 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:00.537626 24494 tablet_service.cc:1467] Processing CreateTablet for tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678 (DEFAULT_TABLE table=test-workload [id=2df34486499a4072ae46eb5f28b49d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:00.537766 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:00.538820 24418 tablet_service.cc:1467] Processing CreateTablet for tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678 (DEFAULT_TABLE table=test-workload [id=2df34486499a4072ae46eb5f28b49d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:00.538849 24569 tablet_service.cc:1467] Processing CreateTablet for tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678 (DEFAULT_TABLE table=test-workload [id=2df34486499a4072ae46eb5f28b49d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:00.538949 24569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:00.538950 24418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:00.539315 24620 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Bootstrap starting.
I20240503 08:11:00.539693 24620 tablet_bootstrap.cc:654] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:00.540285 24623 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Bootstrap starting.
I20240503 08:11:00.540661 24623 tablet_bootstrap.cc:654] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:00.540673 24620 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: No bootstrap required, opened a new log
I20240503 08:11:00.540745 24620 ts_tablet_manager.cc:1367] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:00.540911 24620 raft_consensus.cc:348] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.540992 24620 raft_consensus.cc:374] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:00.541018 24620 raft_consensus.cc:729] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7e68b53494449eaa02c41000022543, State: Initialized, Role: FOLLOWER
I20240503 08:11:00.541060 24622 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81: Bootstrap starting.
I20240503 08:11:00.541077 24620 consensus_queue.cc:260] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.541253 24620 ts_tablet_manager.cc:1398] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.541301 24530 heartbeater.cc:503] Master 127.16.212.254:43563 was elected leader, sending a full tablet report...
I20240503 08:11:00.541417 24622 tablet_bootstrap.cc:654] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:00.541846 24623 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: No bootstrap required, opened a new log
I20240503 08:11:00.541896 24623 ts_tablet_manager.cc:1367] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:00.542037 24623 raft_consensus.cc:348] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.542090 24623 raft_consensus.cc:374] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:00.542104 24623 raft_consensus.cc:729] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9768e075afb4a2e962ab69c85fc526a, State: Initialized, Role: FOLLOWER
I20240503 08:11:00.542137 24623 consensus_queue.cc:260] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.542279 24623 ts_tablet_manager.cc:1398] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:00.542354 24605 heartbeater.cc:503] Master 127.16.212.254:43563 was elected leader, sending a full tablet report...
I20240503 08:11:00.542563 24622 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81: No bootstrap required, opened a new log
I20240503 08:11:00.542650 24622 ts_tablet_manager.cc:1367] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:00.542795 24622 raft_consensus.cc:348] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.542860 24622 raft_consensus.cc:374] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:00.542873 24622 raft_consensus.cc:729] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbcd4f100ef4d95a1f1de6bb4849d81, State: Initialized, Role: FOLLOWER
I20240503 08:11:00.542937 24622 consensus_queue.cc:260] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.543089 24622 ts_tablet_manager.cc:1398] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:00.543180 24454 heartbeater.cc:503] Master 127.16.212.254:43563 was elected leader, sending a full tablet report...
I20240503 08:11:00.656356 24627 raft_consensus.cc:482] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:00.656491 24627 raft_consensus.cc:504] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.656718 24627 leader_election.cc:288] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217), 6bbcd4f100ef4d95a1f1de6bb4849d81 (127.16.212.193:41797)
I20240503 08:11:00.659601 24504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678" candidate_uuid: "e9768e075afb4a2e962ab69c85fc526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7e68b53494449eaa02c41000022543" is_pre_election: true
I20240503 08:11:00.659708 24504 raft_consensus.cc:2454] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9768e075afb4a2e962ab69c85fc526a in term 0.
I20240503 08:11:00.659734 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678" candidate_uuid: "e9768e075afb4a2e962ab69c85fc526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" is_pre_election: true
I20240503 08:11:00.659828 24428 raft_consensus.cc:2454] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9768e075afb4a2e962ab69c85fc526a in term 0.
I20240503 08:11:00.659878 24545 leader_election.cc:302] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca7e68b53494449eaa02c41000022543, e9768e075afb4a2e962ab69c85fc526a; no voters:
I20240503 08:11:00.659977 24627 raft_consensus.cc:2781] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:00.660126 24627 raft_consensus.cc:482] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:00.660161 24627 raft_consensus.cc:3037] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:00.660845 24627 raft_consensus.cc:504] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.660972 24627 leader_election.cc:288] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [CANDIDATE]: Term 1 election: Requested vote from peers ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217), 6bbcd4f100ef4d95a1f1de6bb4849d81 (127.16.212.193:41797)
I20240503 08:11:00.661172 24504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678" candidate_uuid: "e9768e075afb4a2e962ab69c85fc526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca7e68b53494449eaa02c41000022543"
I20240503 08:11:00.661198 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678" candidate_uuid: "e9768e075afb4a2e962ab69c85fc526a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81"
I20240503 08:11:00.661227 24504 raft_consensus.cc:3037] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:00.661257 24428 raft_consensus.cc:3037] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:00.661888 24504 raft_consensus.cc:2454] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9768e075afb4a2e962ab69c85fc526a in term 1.
I20240503 08:11:00.661922 24428 raft_consensus.cc:2454] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9768e075afb4a2e962ab69c85fc526a in term 1.
I20240503 08:11:00.662101 24544 leader_election.cc:302] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbcd4f100ef4d95a1f1de6bb4849d81, e9768e075afb4a2e962ab69c85fc526a; no voters:
I20240503 08:11:00.662207 24627 raft_consensus.cc:2781] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:00.662369 24627 raft_consensus.cc:686] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 LEADER]: Becoming Leader. State: Replica: e9768e075afb4a2e962ab69c85fc526a, State: Running, Role: LEADER
I20240503 08:11:00.662438 24627 consensus_queue.cc:237] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:00.662986 24325 catalog_manager.cc:5484] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a reported cstate change: term changed from 0 to 1, leader changed from <none> to e9768e075afb4a2e962ab69c85fc526a (127.16.212.195). New cstate: current_term: 1 leader_uuid: "e9768e075afb4a2e962ab69c85fc526a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:00.690387 24428 raft_consensus.cc:1261] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 1 FOLLOWER]: Refusing update from remote peer e9768e075afb4a2e962ab69c85fc526a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:00.690552 24504 raft_consensus.cc:1261] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 1 FOLLOWER]: Refusing update from remote peer e9768e075afb4a2e962ab69c85fc526a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:00.690721 24630 consensus_queue.cc:1035] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:00.690848 24627 consensus_queue.cc:1035] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:11.211390 24673 raft_consensus.cc:979] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: : Instructing follower 6bbcd4f100ef4d95a1f1de6bb4849d81 to start an election
I20240503 08:11:11.211474 24673 raft_consensus.cc:1067] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 LEADER]: Signalling peer 6bbcd4f100ef4d95a1f1de6bb4849d81 to start an election
I20240503 08:11:11.211716 24428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678"
dest_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81"
from {username='slave'} at 127.0.0.1:53242
I20240503 08:11:11.211800 24428 raft_consensus.cc:482] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:11.211822 24428 raft_consensus.cc:3037] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:11.212487 24428 raft_consensus.cc:504] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:11.212652 24428 leader_election.cc:288] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [CANDIDATE]: Term 2 election: Requested vote from peers e9768e075afb4a2e962ab69c85fc526a (127.16.212.195:39581), ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217)
I20240503 08:11:11.216208 24504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678" candidate_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" candidate_term: 2 candidate_status { last_received { term: 1 index: 13753 } } ignore_live_leader: true dest_uuid: "ca7e68b53494449eaa02c41000022543"
I20240503 08:11:11.216286 24504 raft_consensus.cc:3037] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:11.216264 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d59ebdbd1e4b83a6a9c3b9b5a89678" candidate_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" candidate_term: 2 candidate_status { last_received { term: 1 index: 13753 } } ignore_live_leader: true dest_uuid: "e9768e075afb4a2e962ab69c85fc526a"
I20240503 08:11:11.216324 24579 raft_consensus.cc:3032] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:11.216363 24579 raft_consensus.cc:729] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9768e075afb4a2e962ab69c85fc526a, State: Running, Role: LEADER
I20240503 08:11:11.216447 24579 consensus_queue.cc:260] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13753, Committed index: 13753, Last appended: 1.13753, Last appended by leader: 13753, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:11.216532 24579 raft_consensus.cc:3037] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:11.216997 24504 raft_consensus.cc:2454] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbcd4f100ef4d95a1f1de6bb4849d81 in term 2.
I20240503 08:11:11.217160 24579 raft_consensus.cc:2454] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbcd4f100ef4d95a1f1de6bb4849d81 in term 2.
I20240503 08:11:11.217218 24394 leader_election.cc:302] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbcd4f100ef4d95a1f1de6bb4849d81, ca7e68b53494449eaa02c41000022543; no voters:
I20240503 08:11:11.217417 24679 raft_consensus.cc:2781] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:11.217562 24679 raft_consensus.cc:686] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 2 LEADER]: Becoming Leader. State: Replica: 6bbcd4f100ef4d95a1f1de6bb4849d81, State: Running, Role: LEADER
I20240503 08:11:11.217617 24679 consensus_queue.cc:237] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13753, Committed index: 13753, Last appended: 1.13753, Last appended by leader: 13753, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:11.218096 24323 catalog_manager.cc:5484] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 reported cstate change: term changed from 1 to 2, leader changed from e9768e075afb4a2e962ab69c85fc526a (127.16.212.195) to 6bbcd4f100ef4d95a1f1de6bb4849d81 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:11.225590 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:11:11.228564 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:11.228679 17235 tablet_replica.cc:330] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: stopping tablet replica
I20240503 08:11:11.228739 17235 raft_consensus.cc:2229] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:11.228816 17235 raft_consensus.cc:2258] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:11.242525 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:11:11.243592 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:11:11.246757 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:11.246901 17235 tablet_replica.cc:330] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: stopping tablet replica
I20240503 08:11:11.246956 17235 raft_consensus.cc:2229] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:11.247009 17235 raft_consensus.cc:2258] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:11.270215 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
W20240503 08:11:11.278231 24391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.212.195:39581: connect: Connection refused (error 111)
W20240503 08:11:11.278676 24394 consensus_peers.cc:485] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 -> Peer ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217): Couldn't send request to peer ca7e68b53494449eaa02c41000022543. Status: Network error: Client connection negotiation failed: client connection to 127.16.212.194:34217: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:11.278676 24391 consensus_peers.cc:485] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 -> Peer e9768e075afb4a2e962ab69c85fc526a (127.16.212.195:39581): Couldn't send request to peer e9768e075afb4a2e962ab69c85fc526a. Status: Network error: Client connection negotiation failed: client connection to 127.16.212.195:39581: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 08:11:12.276494 24687 meta_cache.cc:302] tablet 77d59ebdbd1e4b83a6a9c3b9b5a89678: replica 6bbcd4f100ef4d95a1f1de6bb4849d81 (127.16.212.193:41797) has failed: Timed out: ParticipateInTransaction RPC to 127.16.212.193:41797 timed out after 1.000s (SENT)
I20240503 08:11:12.276834 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:12.279160 17235 server_base.cc:1034] running on GCE node
W20240503 08:11:12.279704 24695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:12.280156 24698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:12.280851 24696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:12.281015 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:12.281051 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:12.281064 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723872281064 us; error 0 us; skew 500 ppm
I20240503 08:11:12.281782 17235 webserver.cc:460] Webserver started at http://127.16.212.194:37981/ using document root <none> and password file <none>
I20240503 08:11:12.281857 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:12.281880 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:11:12.283679 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20240503 08:11:12.284226 24703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.284433 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:12.284493 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "ca7e68b53494449eaa02c41000022543"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:12.284556 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:12.291965 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:12.292150 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:12.292286 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:12.292584 24710 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:11:12.293144 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:12.293207 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:12.293237 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:12.293731 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:12.293761 24710 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Bootstrap starting.
I20240503 08:11:12.293772 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:12.297214 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:34217
I20240503 08:11:12.297430 24768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:34217 every 8 connection(s)
I20240503 08:11:12.300556 24769 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43563
I20240503 08:11:12.300622 24769 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:12.300730 24769 heartbeater.cc:511] Master 127.16.212.254:43563 requested a full tablet report, sending...
I20240503 08:11:12.301016 24323 ts_manager.cc:194] Re-registered known tserver with Master: ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217)
I20240503 08:11:12.301367 24323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46424
I20240503 08:11:13.085427 24710 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Bootstrap replayed 1/3 log segments. Stats: ops{read=4957 overwritten=0 applied=4955 ignored=0} inserts{seen=247700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240503 08:11:13.301949 24769 heartbeater.cc:503] Master 127.16.212.254:43563 was elected leader, sending a full tablet report...
W20240503 08:11:13.608604 24391 consensus_peers.cc:485] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 -> Peer e9768e075afb4a2e962ab69c85fc526a (127.16.212.195:39581): Couldn't send request to peer e9768e075afb4a2e962ab69c85fc526a. Status: Network error: Client connection negotiation failed: client connection to 127.16.212.195:39581: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:13.933439 24710 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Bootstrap replayed 2/3 log segments. Stats: ops{read=9904 overwritten=0 applied=9904 ignored=0} inserts{seen=495150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 08:11:13.992630 24394 consensus_peers.cc:485] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 -> Peer ca7e68b53494449eaa02c41000022543 (127.16.212.194:34217): Couldn't send request to peer ca7e68b53494449eaa02c41000022543. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:14.616299 24710 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Bootstrap replayed 3/3 log segments. Stats: ops{read=13753 overwritten=0 applied=13753 ignored=0} inserts{seen=687600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:14.616716 24710 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Bootstrap complete.
I20240503 08:11:14.621575 24710 ts_tablet_manager.cc:1367] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Time spent bootstrapping tablet: real 2.328s user 1.973s sys 0.352s
I20240503 08:11:14.621750 24710 raft_consensus.cc:348] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:14.621819 24710 raft_consensus.cc:729] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7e68b53494449eaa02c41000022543, State: Initialized, Role: FOLLOWER
I20240503 08:11:14.621870 24710 consensus_queue.cc:260] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13753, Last appended: 1.13753, Last appended by leader: 13753, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:14.622042 24710 ts_tablet_manager.cc:1398] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543: Time spent starting tablet: real 0.000s user 0.003s sys 0.001s
I20240503 08:11:14.622238 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:14.624437 24782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:14.625226 24780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:14.625735 24779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:14.626395 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:14.626513 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:14.626547 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:14.626562 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723874626563 us; error 0 us; skew 500 ppm
I20240503 08:11:14.627379 17235 webserver.cc:460] Webserver started at http://127.16.212.195:37901/ using document root <none> and password file <none>
I20240503 08:11:14.627460 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:14.627483 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:11:14.628113 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:14.628651 24787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:14.628809 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:14.628863 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "e9768e075afb4a2e962ab69c85fc526a"
format_stamp: "Formatted at 2024-05-03 08:11:00 on dist-test-slave-zl2t"
I20240503 08:11:14.628918 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:14.638499 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:14.638711 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:14.638852 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:14.639163 24794 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:11:14.639710 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:14.639757 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:14.639791 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:14.640206 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:14.640237 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:14.640244 24794 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Bootstrap starting.
I20240503 08:11:14.643684 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:39581
I20240503 08:11:14.643754 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:39581 every 8 connection(s)
I20240503 08:11:14.646925 24853 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43563
I20240503 08:11:14.646989 24853 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:14.647092 24853 heartbeater.cc:511] Master 127.16.212.254:43563 requested a full tablet report, sending...
I20240503 08:11:14.647367 24323 ts_manager.cc:194] Re-registered known tserver with Master: e9768e075afb4a2e962ab69c85fc526a (127.16.212.195:39581)
I20240503 08:11:14.647698 24323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46426
W20240503 08:11:14.890287 24864 rpcz_store.cc:267] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.0.1:36650 (ReqId={client: 4146111621c444398804ff4287b34ee0, seq_no=0, attempt_no=0}) took 3612 ms (client timeout 999 ms). Trace:
W20240503 08:11:14.890491 24864 rpcz_store.cc:269] 0503 08:11:11.277652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 08:11:11.277696 (+ 44us) service_pool.cc:225] Handling call
0503 08:11:14.890249 (+3612553us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0503 08:11:11.277833 (+ 0us) participant_op.cc:163] PREPARE: starting
0503 08:11:11.277837 (+ 4us) participant_op.cc:199] PREPARE: finished
0503 08:11:11.277851 (+ 14us) participant_op.cc:212] START. Timestamp: 1714723871277848
0503 08:11:11.277851 (+ 0us) op_driver.cc:331] REPLICATION: starting
0503 08:11:11.277881 (+ 30us) log.cc:842] Serialized 66 byte log entry
0503 08:11:14.889950 (+3612069us) op_driver.cc:444] REPLICATION: finished
0503 08:11:14.890074 (+ 124us) participant_op.cc:265] APPLY: starting
0503 08:11:14.890196 (+ 122us) participant_op.cc:270] APPLY: finished
0503 08:11:14.890203 (+ 7us) log.cc:842] Serialized 15 byte log entry
0503 08:11:14.890244 (+ 41us) participant_op.cc:94] Released txn lock
0503 08:11:14.890244 (+ 0us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":91,"raft.queue_time_us":52,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"replication_time_us":3612094,"spinlock_wait_cycles":33920,"thread_start_us":109,"threads_started":2}]]}
I20240503 08:11:15.356364 24794 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Bootstrap replayed 1/3 log segments. Stats: ops{read=4622 overwritten=0 applied=4621 ignored=0} inserts{seen=231000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:15.648201 24853 heartbeater.cc:503] Master 127.16.212.254:43563 was elected leader, sending a full tablet report...
W20240503 08:11:15.940418 24391 consensus_peers.cc:485] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 -> Peer e9768e075afb4a2e962ab69c85fc526a (127.16.212.195:39581): Couldn't send request to peer e9768e075afb4a2e962ab69c85fc526a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20240503 08:11:16.134485 24794 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Bootstrap replayed 2/3 log segments. Stats: ops{read=9243 overwritten=0 applied=9240 ignored=0} inserts{seen=461950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:11:16.931216 24794 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Bootstrap replayed 3/3 log segments. Stats: ops{read=13753 overwritten=0 applied=13753 ignored=0} inserts{seen=687600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:16.931650 24794 tablet_bootstrap.cc:492] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Bootstrap complete.
I20240503 08:11:16.936652 24794 ts_tablet_manager.cc:1367] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Time spent bootstrapping tablet: real 2.296s user 1.990s sys 0.304s
I20240503 08:11:16.936873 24794 raft_consensus.cc:348] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:16.936949 24794 raft_consensus.cc:729] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9768e075afb4a2e962ab69c85fc526a, State: Initialized, Role: FOLLOWER
I20240503 08:11:16.937018 24794 consensus_queue.cc:260] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13753, Last appended: 1.13753, Last appended by leader: 13753, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9768e075afb4a2e962ab69c85fc526a" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 39581 } } peers { permanent_uuid: "ca7e68b53494449eaa02c41000022543" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 34217 } } peers { permanent_uuid: "6bbcd4f100ef4d95a1f1de6bb4849d81" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41797 } }
I20240503 08:11:16.937188 24794 ts_tablet_manager.cc:1398] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20240503 08:11:17.450073 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:11:17.452558 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:17.452694 17235 tablet_replica.cc:330] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81: stopping tablet replica
I20240503 08:11:17.452749 17235 raft_consensus.cc:2229] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.452836 17235 raft_consensus.cc:2258] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P 6bbcd4f100ef4d95a1f1de6bb4849d81 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.466463 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:11:17.467692 17235 tablet_server.cc:178] TabletServer@127.16.212.194:34217 shutting down...
I20240503 08:11:17.469870 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:17.469997 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:11:17.470048 17235 raft_consensus.cc:2229] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.470115 17235 raft_consensus.cc:2258] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P ca7e68b53494449eaa02c41000022543 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.491500 17235 tablet_server.cc:194] TabletServer@127.16.212.194:34217 shutdown complete.
I20240503 08:11:17.492595 17235 tablet_server.cc:178] TabletServer@127.16.212.195:39581 shutting down...
I20240503 08:11:17.494962 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:17.495087 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:11:17.495163 17235 raft_consensus.cc:2229] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:17.495208 17235 raft_consensus.cc:2258] T 77d59ebdbd1e4b83a6a9c3b9b5a89678 P e9768e075afb4a2e962ab69c85fc526a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.508459 17235 tablet_server.cc:194] TabletServer@127.16.212.195:39581 shutdown complete.
I20240503 08:11:17.509408 17235 master.cc:537] Master@127.16.212.254:43563 shutting down...
I20240503 08:11:17.511893 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:17.511981 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:17.512015 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1d27af3660fd4a7b8e6292653fd1e03e: stopping tablet replica
I20240503 08:11:17.523782 17235 master.cc:552] Master@127.16.212.254:43563 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority (17195 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap
I20240503 08:11:17.543068 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:43847
I20240503 08:11:17.543233 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.545362 24872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.545887 24873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.546478 24875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.547348 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:17.547466 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.547493 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.547524 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723877547524 us; error 0 us; skew 500 ppm
I20240503 08:11:17.553304 17235 webserver.cc:460] Webserver started at http://127.16.212.254:36381/ using document root <none> and password file <none>
I20240503 08:11:17.553418 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.553469 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.553522 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.553932 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "50201e51cbaf4f6592a2a57fa3f37fa2"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.554855 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:17.555369 24880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.555549 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:17.555611 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "50201e51cbaf4f6592a2a57fa3f37fa2"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.555660 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.584915 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.585130 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.587870 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:43847
I20240503 08:11:17.599896 24932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:43847 every 8 connection(s)
I20240503 08:11:17.600029 24933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.601112 24933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2: Bootstrap starting.
I20240503 08:11:17.601346 24933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.602042 24933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2: No bootstrap required, opened a new log
I20240503 08:11:17.602177 24933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" member_type: VOTER }
I20240503 08:11:17.602242 24933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.602264 24933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50201e51cbaf4f6592a2a57fa3f37fa2, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.602325 24933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" member_type: VOTER }
I20240503 08:11:17.602367 24933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:17.602391 24933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:17.602416 24933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.603101 24933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" member_type: VOTER }
I20240503 08:11:17.603165 24933 leader_election.cc:302] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50201e51cbaf4f6592a2a57fa3f37fa2; no voters:
I20240503 08:11:17.603266 24933 leader_election.cc:288] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:17.603330 24936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.603425 24933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:17.603463 24936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 50201e51cbaf4f6592a2a57fa3f37fa2, State: Running, Role: LEADER
I20240503 08:11:17.603509 24936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" member_type: VOTER }
I20240503 08:11:17.603672 24938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50201e51cbaf4f6592a2a57fa3f37fa2. Latest consensus state: current_term: 1 leader_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" member_type: VOTER } }
I20240503 08:11:17.603688 24937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50201e51cbaf4f6592a2a57fa3f37fa2" member_type: VOTER } }
I20240503 08:11:17.603828 24937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.603809 24938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:17.604107 24943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:17.604249 24943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:17.604737 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:17.604970 24943 catalog_manager.cc:1347] Generated new cluster ID: 996272bb817e4dba95a0c82b1d9d1a3b
I20240503 08:11:17.605017 24943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:17.610019 24943 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:17.610426 24943 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:17.615936 24943 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2: Generated new TSK 0
I20240503 08:11:17.616007 24943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:17.620407 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.622422 24957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.622900 24954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.623735 24955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.624514 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:17.624593 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.624619 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.624631 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723877624631 us; error 0 us; skew 500 ppm
I20240503 08:11:17.625290 17235 webserver.cc:460] Webserver started at http://127.16.212.193:33125/ using document root <none> and password file <none>
I20240503 08:11:17.625363 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.625404 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.625440 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.625681 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "705113ccef9e44d7a0913e9f532a1084"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.626529 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:17.627061 24962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.627213 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.627257 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "705113ccef9e44d7a0913e9f532a1084"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.627310 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.642081 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.642321 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.642490 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.642728 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.642760 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.642792 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.642818 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.645471 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:40849
I20240503 08:11:17.645530 25025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:40849 every 8 connection(s)
I20240503 08:11:17.645907 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.647984 25031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.648664 25033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.649436 25030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.650852 25026 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43847
I20240503 08:11:17.650918 25026 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.651047 25026 heartbeater.cc:511] Master 127.16.212.254:43847 requested a full tablet report, sending...
I20240503 08:11:17.651335 24897 ts_manager.cc:194] Registered new tserver with Master: 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849)
I20240503 08:11:17.651430 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:17.651573 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:11:17.652334 24897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38762
W20240503 08:11:17.651604 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.652541 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723877652541 us; error 0 us; skew 500 ppm
I20240503 08:11:17.653494 17235 webserver.cc:460] Webserver started at http://127.16.212.194:45451/ using document root <none> and password file <none>
I20240503 08:11:17.653599 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.653647 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.653702 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.654106 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "bfc1ed9303764aada8541d8bbc31ed09"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.655395 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:11:17.656060 25038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.656231 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:17.656306 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "bfc1ed9303764aada8541d8bbc31ed09"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.656355 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.671666 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.671902 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.672057 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.672257 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.672284 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.672318 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.672344 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.675101 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:43021
I20240503 08:11:17.675170 25101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:43021 every 8 connection(s)
I20240503 08:11:17.675455 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:17.677173 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:17.677809 25108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.678556 25102 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43847
I20240503 08:11:17.678630 25102 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.678654 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:17.678755 25102 heartbeater.cc:511] Master 127.16.212.254:43847 requested a full tablet report, sending...
I20240503 08:11:17.679000 24897 ts_manager.cc:194] Registered new tserver with Master: bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
W20240503 08:11:17.679234 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:17.679404 24897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38772
I20240503 08:11:17.679433 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:17.679459 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:17.679472 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723877679472 us; error 0 us; skew 500 ppm
I20240503 08:11:17.680145 17235 webserver.cc:460] Webserver started at http://127.16.212.195:46321/ using document root <none> and password file <none>
I20240503 08:11:17.680213 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:17.680248 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:17.680305 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:17.680610 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "c5c48e7baba24a348bb082bf89fb4f29"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.681490 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:17.681917 25113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.682073 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.682113 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "c5c48e7baba24a348bb082bf89fb4f29"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:17.682140 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:17.711637 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:17.711850 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:17.712033 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:17.712253 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:17.712284 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.712308 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:17.712337 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:17.715588 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:40417
I20240503 08:11:17.715665 25176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:40417 every 8 connection(s)
I20240503 08:11:17.718750 25177 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43847
I20240503 08:11:17.718813 25177 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:17.718894 25177 heartbeater.cc:511] Master 127.16.212.254:43847 requested a full tablet report, sending...
I20240503 08:11:17.719151 24897 ts_manager.cc:194] Registered new tserver with Master: c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417)
I20240503 08:11:17.719184 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003412271s
I20240503 08:11:17.719517 24897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38788
I20240503 08:11:17.719527 17235 test_util.cc:268] Using random seed: 1134464487
I20240503 08:11:17.724084 24897 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38802:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:11:17.724205 24897 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:17.728807 24990 tablet_service.cc:1467] Processing CreateTablet for tablet df9b5dd5e29948169ece9901d6742584 (DEFAULT_TABLE table=test-workload [id=d54fe28d8fe64edc8fa6dccf5595dada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.728956 24990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9b5dd5e29948169ece9901d6742584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.729137 25141 tablet_service.cc:1467] Processing CreateTablet for tablet df9b5dd5e29948169ece9901d6742584 (DEFAULT_TABLE table=test-workload [id=d54fe28d8fe64edc8fa6dccf5595dada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.729249 25141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9b5dd5e29948169ece9901d6742584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.729312 25066 tablet_service.cc:1467] Processing CreateTablet for tablet df9b5dd5e29948169ece9901d6742584 (DEFAULT_TABLE table=test-workload [id=d54fe28d8fe64edc8fa6dccf5595dada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:17.729419 25066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df9b5dd5e29948169ece9901d6742584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:17.730470 25193 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Bootstrap starting.
I20240503 08:11:17.730722 25192 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Bootstrap starting.
I20240503 08:11:17.730818 25194 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Bootstrap starting.
I20240503 08:11:17.730988 25193 tablet_bootstrap.cc:654] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.731256 25194 tablet_bootstrap.cc:654] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.731259 25192 tablet_bootstrap.cc:654] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:17.731657 25193 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: No bootstrap required, opened a new log
I20240503 08:11:17.731766 25193 ts_tablet_manager.cc:1367] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:17.731803 25194 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: No bootstrap required, opened a new log
I20240503 08:11:17.731856 25194 ts_tablet_manager.cc:1367] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:17.731916 25193 raft_consensus.cc:348] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.731986 25193 raft_consensus.cc:374] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.732002 25193 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705113ccef9e44d7a0913e9f532a1084, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.732003 25194 raft_consensus.cc:348] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.732090 25194 raft_consensus.cc:374] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.732080 25193 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.732136 25194 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc1ed9303764aada8541d8bbc31ed09, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.732190 25194 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.732255 25193 ts_tablet_manager.cc:1398] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:17.732288 25026 heartbeater.cc:503] Master 127.16.212.254:43847 was elected leader, sending a full tablet report...
I20240503 08:11:17.732344 25194 ts_tablet_manager.cc:1398] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:17.732420 25102 heartbeater.cc:503] Master 127.16.212.254:43847 was elected leader, sending a full tablet report...
I20240503 08:11:17.732573 25192 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: No bootstrap required, opened a new log
I20240503 08:11:17.732611 25192 ts_tablet_manager.cc:1367] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:17.732748 25192 raft_consensus.cc:348] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.732800 25192 raft_consensus.cc:374] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:17.732820 25192 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c48e7baba24a348bb082bf89fb4f29, State: Initialized, Role: FOLLOWER
I20240503 08:11:17.732857 25192 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.733017 25192 ts_tablet_manager.cc:1398] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:17.733204 25177 heartbeater.cc:503] Master 127.16.212.254:43847 was elected leader, sending a full tablet report...
I20240503 08:11:17.798691 25199 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:17.798803 25199 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.799046 25199 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417), 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849)
I20240503 08:11:17.801800 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "bfc1ed9303764aada8541d8bbc31ed09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705113ccef9e44d7a0913e9f532a1084" is_pre_election: true
I20240503 08:11:17.801905 25000 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc1ed9303764aada8541d8bbc31ed09 in term 0.
I20240503 08:11:17.801932 25151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "bfc1ed9303764aada8541d8bbc31ed09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c48e7baba24a348bb082bf89fb4f29" is_pre_election: true
I20240503 08:11:17.802029 25151 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc1ed9303764aada8541d8bbc31ed09 in term 0.
I20240503 08:11:17.802073 25041 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 705113ccef9e44d7a0913e9f532a1084, bfc1ed9303764aada8541d8bbc31ed09; no voters:
I20240503 08:11:17.802160 25199 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:17.802201 25199 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:17.802232 25199 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.802907 25199 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.803035 25199 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 1 election: Requested vote from peers c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417), 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849)
I20240503 08:11:17.803156 25151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "bfc1ed9303764aada8541d8bbc31ed09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5c48e7baba24a348bb082bf89fb4f29"
I20240503 08:11:17.803203 25000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "bfc1ed9303764aada8541d8bbc31ed09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "705113ccef9e44d7a0913e9f532a1084"
I20240503 08:11:17.803206 25151 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.803297 25000 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:17.803967 25151 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc1ed9303764aada8541d8bbc31ed09 in term 1.
I20240503 08:11:17.803987 25000 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc1ed9303764aada8541d8bbc31ed09 in term 1.
I20240503 08:11:17.804114 25042 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfc1ed9303764aada8541d8bbc31ed09, c5c48e7baba24a348bb082bf89fb4f29; no voters:
I20240503 08:11:17.804209 25199 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:17.804436 25199 raft_consensus.cc:686] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 LEADER]: Becoming Leader. State: Replica: bfc1ed9303764aada8541d8bbc31ed09, State: Running, Role: LEADER
I20240503 08:11:17.804553 25199 consensus_queue.cc:237] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:17.805212 24896 catalog_manager.cc:5484] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194). New cstate: current_term: 1 leader_uuid: "bfc1ed9303764aada8541d8bbc31ed09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:17.830539 25000 raft_consensus.cc:1261] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 1 FOLLOWER]: Refusing update from remote peer bfc1ed9303764aada8541d8bbc31ed09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:17.830629 25151 raft_consensus.cc:1261] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 1 FOLLOWER]: Refusing update from remote peer bfc1ed9303764aada8541d8bbc31ed09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:17.830830 25199 consensus_queue.cc:1035] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:17.830937 25202 consensus_queue.cc:1035] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:28.262147 25241 raft_consensus.cc:979] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: : Instructing follower 705113ccef9e44d7a0913e9f532a1084 to start an election
I20240503 08:11:28.262238 25241 raft_consensus.cc:1067] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 LEADER]: Signalling peer 705113ccef9e44d7a0913e9f532a1084 to start an election
I20240503 08:11:28.262545 24999 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584"
dest_uuid: "705113ccef9e44d7a0913e9f532a1084"
from {username='slave'} at 127.0.0.1:33942
I20240503 08:11:28.262634 24999 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:28.262655 24999 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:28.263334 24999 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:28.263473 24999 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 2 election: Requested vote from peers c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417), bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
I20240503 08:11:28.267123 25150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "705113ccef9e44d7a0913e9f532a1084" candidate_term: 2 candidate_status { last_received { term: 1 index: 13433 } } ignore_live_leader: true dest_uuid: "c5c48e7baba24a348bb082bf89fb4f29"
I20240503 08:11:28.267139 25076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "705113ccef9e44d7a0913e9f532a1084" candidate_term: 2 candidate_status { last_received { term: 1 index: 13433 } } ignore_live_leader: true dest_uuid: "bfc1ed9303764aada8541d8bbc31ed09"
I20240503 08:11:28.267221 25150 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:28.267228 25076 raft_consensus.cc:3032] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:11:28.267272 25076 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bfc1ed9303764aada8541d8bbc31ed09, State: Running, Role: LEADER
I20240503 08:11:28.267344 25076 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13433, Committed index: 13433, Last appended: 1.13433, Last appended by leader: 13433, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:28.267452 25076 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:28.267968 25150 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705113ccef9e44d7a0913e9f532a1084 in term 2.
I20240503 08:11:28.268088 25076 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705113ccef9e44d7a0913e9f532a1084 in term 2.
I20240503 08:11:28.268198 24966 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 705113ccef9e44d7a0913e9f532a1084, c5c48e7baba24a348bb082bf89fb4f29; no voters:
I20240503 08:11:28.268424 25255 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:28.268550 25255 raft_consensus.cc:686] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 LEADER]: Becoming Leader. State: Replica: 705113ccef9e44d7a0913e9f532a1084, State: Running, Role: LEADER
I20240503 08:11:28.268604 25255 consensus_queue.cc:237] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13433, Committed index: 13433, Last appended: 1.13433, Last appended by leader: 13433, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:28.269057 24896 catalog_manager.cc:5484] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 reported cstate change: term changed from 1 to 2, leader changed from bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194) to 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "705113ccef9e44d7a0913e9f532a1084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:28.280730 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:11:28.283805 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:28.283946 17235 tablet_replica.cc:330] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: stopping tablet replica
I20240503 08:11:28.284006 17235 raft_consensus.cc:2229] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:11:28.284088 17235 pending_rounds.cc:62] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Trying to abort 1 pending ops.
I20240503 08:11:28.284116 17235 pending_rounds.cc:68] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Aborting op as it isn't in flight: id { term: 2 index: 13434 } timestamp: 7023509046348685312 op_type: NO_OP noop_request { }
I20240503 08:11:28.284169 17235 raft_consensus.cc:2866] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:11:28.284199 17235 raft_consensus.cc:2258] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 08:11:28.290776 25263 meta_cache.cc:302] tablet df9b5dd5e29948169ece9901d6742584: replica 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849) has failed: Network error: Client connection negotiation failed: client connection to 127.16.212.193:40849: connect: Connection refused (error 111)
W20240503 08:11:28.298527 25141 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51796: Illegal state: replica c5c48e7baba24a348bb082bf89fb4f29 is not leader of this config: current role FOLLOWER
I20240503 08:11:28.299469 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:11:28.300419 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:28.302578 25272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:28.303380 25270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:28.304226 17235 server_base.cc:1034] running on GCE node
W20240503 08:11:28.304735 25269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:28.304930 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:28.304988 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:28.305013 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723888305013 us; error 0 us; skew 500 ppm
I20240503 08:11:28.305943 17235 webserver.cc:460] Webserver started at http://127.16.212.193:33125/ using document root <none> and password file <none>
I20240503 08:11:28.306037 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:28.306075 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:11:28.306798 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:28.307478 25277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20240503 08:11:28.307611 25066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44124: Illegal state: replica bfc1ed9303764aada8541d8bbc31ed09 is not leader of this config: current role FOLLOWER
I20240503 08:11:28.307782 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:28.307840 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "705113ccef9e44d7a0913e9f532a1084"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:28.307912 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:28.316917 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:28.317095 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:28.317214 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:28.317471 25284 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:11:28.318074 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:28.318113 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:28.318162 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:28.318540 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:28.318571 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:28.318583 25284 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Bootstrap starting.
I20240503 08:11:28.322530 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:40849
W20240503 08:11:28.325153 25141 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51796: Illegal state: replica c5c48e7baba24a348bb082bf89fb4f29 is not leader of this config: current role FOLLOWER
I20240503 08:11:28.328779 25342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:40849 every 8 connection(s)
I20240503 08:11:28.332553 25343 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43847
I20240503 08:11:28.332628 25343 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:28.332742 25343 heartbeater.cc:511] Master 127.16.212.254:43847 requested a full tablet report, sending...
W20240503 08:11:28.332784 25066 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44124: Illegal state: replica bfc1ed9303764aada8541d8bbc31ed09 is not leader of this config: current role FOLLOWER
I20240503 08:11:28.333108 24896 ts_manager.cc:194] Re-registered known tserver with Master: 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849)
I20240503 08:11:28.333483 24896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37482
I20240503 08:11:29.102397 25284 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Bootstrap replayed 1/3 log segments. Stats: ops{read=4942 overwritten=0 applied=4940 ignored=0} inserts{seen=246950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240503 08:11:29.334013 25343 heartbeater.cc:503] Master 127.16.212.254:43847 was elected leader, sending a full tablet report...
I20240503 08:11:29.877810 25349 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:29.877909 25349 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:29.878106 25349 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849), bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
I20240503 08:11:29.881114 25076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "c5c48e7baba24a348bb082bf89fb4f29" candidate_term: 3 candidate_status { last_received { term: 1 index: 13433 } } ignore_live_leader: false dest_uuid: "bfc1ed9303764aada8541d8bbc31ed09" is_pre_election: true
I20240503 08:11:29.881217 25076 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5c48e7baba24a348bb082bf89fb4f29 in term 2.
I20240503 08:11:29.881259 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "c5c48e7baba24a348bb082bf89fb4f29" candidate_term: 3 candidate_status { last_received { term: 1 index: 13433 } } ignore_live_leader: false dest_uuid: "705113ccef9e44d7a0913e9f532a1084" is_pre_election: true
I20240503 08:11:29.881472 25117 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfc1ed9303764aada8541d8bbc31ed09, c5c48e7baba24a348bb082bf89fb4f29; no voters:
I20240503 08:11:29.881574 25349 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240503 08:11:29.881582 25116 leader_election.cc:341] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:29.881637 25349 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:29.881667 25349 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:29.882599 25349 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:29.882759 25349 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [CANDIDATE]: Term 3 election: Requested vote from peers 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849), bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
I20240503 08:11:29.882905 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "c5c48e7baba24a348bb082bf89fb4f29" candidate_term: 3 candidate_status { last_received { term: 1 index: 13433 } } ignore_live_leader: false dest_uuid: "705113ccef9e44d7a0913e9f532a1084"
I20240503 08:11:29.882951 25076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "c5c48e7baba24a348bb082bf89fb4f29" candidate_term: 3 candidate_status { last_received { term: 1 index: 13433 } } ignore_live_leader: false dest_uuid: "bfc1ed9303764aada8541d8bbc31ed09"
I20240503 08:11:29.883008 25076 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 2 FOLLOWER]: Advancing to term 3
W20240503 08:11:29.883093 25116 leader_election.cc:341] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:29.883702 25076 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5c48e7baba24a348bb082bf89fb4f29 in term 3.
I20240503 08:11:29.883862 25117 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfc1ed9303764aada8541d8bbc31ed09, c5c48e7baba24a348bb082bf89fb4f29; no voters: 705113ccef9e44d7a0913e9f532a1084
I20240503 08:11:29.883945 25349 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:11:29.884091 25349 raft_consensus.cc:686] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 LEADER]: Becoming Leader. State: Replica: c5c48e7baba24a348bb082bf89fb4f29, State: Running, Role: LEADER
I20240503 08:11:29.884145 25349 consensus_queue.cc:237] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13433, Committed index: 13433, Last appended: 1.13433, Last appended by leader: 13433, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:29.884665 24896 catalog_manager.cc:5484] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 reported cstate change: term changed from 2 to 3, leader changed from 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193) to c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195). New cstate: current_term: 3 leader_uuid: "c5c48e7baba24a348bb082bf89fb4f29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:29.984045 25284 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Bootstrap replayed 2/3 log segments. Stats: ops{read=9903 overwritten=0 applied=9898 ignored=0} inserts{seen=494850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20240503 08:11:30.285342 25076 raft_consensus.cc:1261] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Refusing update from remote peer c5c48e7baba24a348bb082bf89fb4f29: Log matching property violated. Preceding OpId in replica: term: 1 index: 13433. Preceding OpId from leader: term: 3 index: 13434. (index mismatch)
I20240503 08:11:30.285640 25349 consensus_queue.cc:1035] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13434, Last known committed idx: 13433, Time since last communication: 0.000s
W20240503 08:11:30.287118 25116 consensus_peers.cc:485] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 -> Peer 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849): Couldn't send request to peer 705113ccef9e44d7a0913e9f532a1084. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:30.625429 25284 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Bootstrap replayed 3/3 log segments. Stats: ops{read=13434 overwritten=0 applied=13433 ignored=0} inserts{seen=671600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:30.625829 25284 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Bootstrap complete.
I20240503 08:11:30.630554 25284 ts_tablet_manager.cc:1367] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Time spent bootstrapping tablet: real 2.312s user 1.964s sys 0.343s
I20240503 08:11:30.630757 25284 raft_consensus.cc:348] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:30.630836 25284 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705113ccef9e44d7a0913e9f532a1084, State: Initialized, Role: FOLLOWER
I20240503 08:11:30.630883 25284 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13433, Last appended: 2.13434, Last appended by leader: 13434, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:30.631050 25284 ts_tablet_manager.cc:1398] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20240503 08:11:30.691613 25307 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38606: Illegal state: replica 705113ccef9e44d7a0913e9f532a1084 is not leader of this config: current role FOLLOWER
I20240503 08:11:30.861431 25317 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:30.862241 25317 pending_rounds.cc:76] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Aborting all ops after (but not including) 13433
I20240503 08:11:30.862284 25317 pending_rounds.cc:97] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084: Aborting uncommitted NO_OP operation due to leader change: 2.13434
I20240503 08:11:30.862324 25317 raft_consensus.cc:2866] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:11:31.131512 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
W20240503 08:11:31.132086 25117 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 8 similar messages]
W20240503 08:11:31.133734 25117 consensus_peers.cc:485] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 -> Peer bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021): Couldn't send request to peer bfc1ed9303764aada8541d8bbc31ed09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:31.134470 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:31.134776 17235 tablet_replica.cc:330] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: stopping tablet replica
I20240503 08:11:31.134863 17235 raft_consensus.cc:2229] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:31.134912 17235 raft_consensus.cc:2258] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:31.140920 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:11:31.142246 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:31.145160 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:31.146108 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:31.146389 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:31.147217 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:31.147364 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:31.147405 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:31.147426 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723891147426 us; error 0 us; skew 500 ppm
I20240503 08:11:31.148460 17235 webserver.cc:460] Webserver started at http://127.16.212.194:45451/ using document root <none> and password file <none>
I20240503 08:11:31.148559 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:31.148602 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:11:31.149335 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:31.149942 25382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:31.150166 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:31.150313 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "bfc1ed9303764aada8541d8bbc31ed09"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:31.150365 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:31.160665 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:31.160912 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:31.161135 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:31.161541 25389 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:11:31.162397 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:31.162487 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:31.162544 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:31.163069 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:31.163192 25389 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Bootstrap starting.
I20240503 08:11:31.163260 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:31.167097 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:43021
I20240503 08:11:31.174016 25447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:43021 every 8 connection(s)
I20240503 08:11:31.180114 25448 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43847
I20240503 08:11:31.180320 25448 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:31.180514 25448 heartbeater.cc:511] Master 127.16.212.254:43847 requested a full tablet report, sending...
I20240503 08:11:31.180989 24896 ts_manager.cc:194] Re-registered known tserver with Master: bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
I20240503 08:11:31.181510 24896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37486
I20240503 08:11:31.945883 25389 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Bootstrap replayed 1/3 log segments. Stats: ops{read=4621 overwritten=0 applied=4620 ignored=0} inserts{seen=230950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:32.182129 25448 heartbeater.cc:503] Master 127.16.212.254:43847 was elected leader, sending a full tablet report...
I20240503 08:11:32.810199 25389 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Bootstrap replayed 2/3 log segments. Stats: ops{read=9241 overwritten=0 applied=9240 ignored=0} inserts{seen=461950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20240503 08:11:33.527545 25117 consensus_peers.cc:485] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 -> Peer bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021): Couldn't send request to peer bfc1ed9303764aada8541d8bbc31ed09. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:33.619225 25389 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Bootstrap replayed 3/3 log segments. Stats: ops{read=13822 overwritten=0 applied=13822 ignored=0} inserts{seen=671600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:33.619710 25389 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Bootstrap complete.
I20240503 08:11:33.626540 25389 ts_tablet_manager.cc:1367] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Time spent bootstrapping tablet: real 2.463s user 2.129s sys 0.324s
I20240503 08:11:33.626734 25389 raft_consensus.cc:348] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:33.626833 25389 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc1ed9303764aada8541d8bbc31ed09, State: Initialized, Role: FOLLOWER
I20240503 08:11:33.626883 25389 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13822, Last appended: 3.13822, Last appended by leader: 13822, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:33.627091 25389 ts_tablet_manager.cc:1398] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:34.127380 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:11:34.130296 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:34.130457 17235 tablet_replica.cc:330] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: stopping tablet replica
I20240503 08:11:34.130522 17235 raft_consensus.cc:2229] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:11:34.131156 17235 raft_consensus.cc:2258] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 08:11:34.134399 17235 mvcc.cc:119] aborting op with timestamp 7023509070345396224 in state 0; MVCC is closed
W20240503 08:11:34.138249 25264 meta_cache.cc:302] tablet df9b5dd5e29948169ece9901d6742584: replica c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417) has failed: Network error: Client connection negotiation failed: client connection to 127.16.212.195:40417: connect: Connection refused (error 111)
W20240503 08:11:34.146008 25307 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38606: Illegal state: replica 705113ccef9e44d7a0913e9f532a1084 is not leader of this config: current role FOLLOWER
I20240503 08:11:34.147003 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:11:34.148474 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:34.150681 25465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:34.151428 25466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:34.151986 25468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:34.152678 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:34.152776 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:34.152809 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:34.152833 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723894152833 us; error 0 us; skew 500 ppm
W20240503 08:11:34.154331 25408 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44152: Illegal state: replica bfc1ed9303764aada8541d8bbc31ed09 is not leader of this config: current role FOLLOWER
I20240503 08:11:34.158002 17235 webserver.cc:460] Webserver started at http://127.16.212.195:46321/ using document root <none> and password file <none>
I20240503 08:11:34.158083 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:34.158114 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:11:34.158704 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:34.159150 25474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:34.159281 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:34.159332 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "c5c48e7baba24a348bb082bf89fb4f29"
format_stamp: "Formatted at 2024-05-03 08:11:17 on dist-test-slave-zl2t"
I20240503 08:11:34.159384 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 08:11:34.166858 25307 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38606: Illegal state: replica 705113ccef9e44d7a0913e9f532a1084 is not leader of this config: current role FOLLOWER
W20240503 08:11:34.176530 25408 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44152: Illegal state: replica bfc1ed9303764aada8541d8bbc31ed09 is not leader of this config: current role FOLLOWER
I20240503 08:11:34.180045 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:34.180255 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:34.180416 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:34.180697 25481 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:11:34.181314 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:11:34.181366 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:34.181406 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:11:34.181825 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:11:34.181865 25481 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Bootstrap starting.
I20240503 08:11:34.181855 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:34.185295 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:40417
I20240503 08:11:34.187227 25539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:40417 every 8 connection(s)
I20240503 08:11:34.197428 25540 heartbeater.cc:347] Connected to a master server at 127.16.212.254:43847
I20240503 08:11:34.197501 25540 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:34.197592 25540 heartbeater.cc:511] Master 127.16.212.254:43847 requested a full tablet report, sending...
I20240503 08:11:34.197872 24896 ts_manager.cc:194] Re-registered known tserver with Master: c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417)
I20240503 08:11:34.198385 24896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37500
I20240503 08:11:34.964969 25481 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Bootstrap replayed 1/3 log segments. Stats: ops{read=4946 overwritten=0 applied=4944 ignored=0} inserts{seen=247150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240503 08:11:35.198997 25540 heartbeater.cc:503] Master 127.16.212.254:43847 was elected leader, sending a full tablet report...
I20240503 08:11:35.259907 25546 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:35.260027 25546 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:35.260238 25546 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417), 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193:40849)
I20240503 08:11:35.263296 25317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "bfc1ed9303764aada8541d8bbc31ed09" candidate_term: 4 candidate_status { last_received { term: 3 index: 13822 } } ignore_live_leader: false dest_uuid: "705113ccef9e44d7a0913e9f532a1084" is_pre_election: true
I20240503 08:11:35.263535 25514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "bfc1ed9303764aada8541d8bbc31ed09" candidate_term: 4 candidate_status { last_received { term: 3 index: 13822 } } ignore_live_leader: false dest_uuid: "c5c48e7baba24a348bb082bf89fb4f29" is_pre_election: true
W20240503 08:11:35.263759 25386 leader_election.cc:341] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:35.263823 25386 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfc1ed9303764aada8541d8bbc31ed09; no voters: 705113ccef9e44d7a0913e9f532a1084, c5c48e7baba24a348bb082bf89fb4f29
I20240503 08:11:35.263938 25546 raft_consensus.cc:2726] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 08:11:35.628417 25551 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5c48e7baba24a348bb082bf89fb4f29)
I20240503 08:11:35.628515 25551 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:35.629240 25551 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417), bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
I20240503 08:11:35.631744 25514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "705113ccef9e44d7a0913e9f532a1084" candidate_term: 4 candidate_status { last_received { term: 3 index: 14642 } } ignore_live_leader: false dest_uuid: "c5c48e7baba24a348bb082bf89fb4f29" is_pre_election: true
W20240503 08:11:35.631983 25281 leader_election.cc:341] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:35.632339 25416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "705113ccef9e44d7a0913e9f532a1084" candidate_term: 4 candidate_status { last_received { term: 3 index: 14642 } } ignore_live_leader: false dest_uuid: "bfc1ed9303764aada8541d8bbc31ed09" is_pre_election: true
I20240503 08:11:35.632426 25416 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 705113ccef9e44d7a0913e9f532a1084 in term 3.
I20240503 08:11:35.632565 25281 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 705113ccef9e44d7a0913e9f532a1084, bfc1ed9303764aada8541d8bbc31ed09; no voters: c5c48e7baba24a348bb082bf89fb4f29
I20240503 08:11:35.632675 25551 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 08:11:35.632725 25551 raft_consensus.cc:482] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5c48e7baba24a348bb082bf89fb4f29)
I20240503 08:11:35.632752 25551 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:11:35.633677 25551 raft_consensus.cc:504] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:35.633803 25551 leader_election.cc:288] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 4 election: Requested vote from peers c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417), bfc1ed9303764aada8541d8bbc31ed09 (127.16.212.194:43021)
I20240503 08:11:35.634065 25514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "705113ccef9e44d7a0913e9f532a1084" candidate_term: 4 candidate_status { last_received { term: 3 index: 14642 } } ignore_live_leader: false dest_uuid: "c5c48e7baba24a348bb082bf89fb4f29"
I20240503 08:11:35.634074 25416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df9b5dd5e29948169ece9901d6742584" candidate_uuid: "705113ccef9e44d7a0913e9f532a1084" candidate_term: 4 candidate_status { last_received { term: 3 index: 14642 } } ignore_live_leader: false dest_uuid: "bfc1ed9303764aada8541d8bbc31ed09"
I20240503 08:11:35.634167 25416 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 3 FOLLOWER]: Advancing to term 4
W20240503 08:11:35.634258 25281 leader_election.cc:341] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 4 election: Tablet error from VoteRequest() call to peer c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417): Illegal state: must be running to vote when last-logged opid is not known
I20240503 08:11:35.634917 25416 raft_consensus.cc:2454] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705113ccef9e44d7a0913e9f532a1084 in term 4.
I20240503 08:11:35.635097 25281 leader_election.cc:302] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 705113ccef9e44d7a0913e9f532a1084, bfc1ed9303764aada8541d8bbc31ed09; no voters: c5c48e7baba24a348bb082bf89fb4f29
I20240503 08:11:35.635212 25551 raft_consensus.cc:2781] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:11:35.635334 25551 raft_consensus.cc:686] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 4 LEADER]: Becoming Leader. State: Replica: 705113ccef9e44d7a0913e9f532a1084, State: Running, Role: LEADER
I20240503 08:11:35.635391 25551 consensus_queue.cc:237] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14642, Committed index: 14642, Last appended: 3.14642, Last appended by leader: 14642, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:35.635913 24896 catalog_manager.cc:5484] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 reported cstate change: term changed from 3 to 4, leader changed from c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195) to 705113ccef9e44d7a0913e9f532a1084 (127.16.212.193). New cstate: current_term: 4 leader_uuid: "705113ccef9e44d7a0913e9f532a1084" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:35.812420 25481 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Bootstrap replayed 2/3 log segments. Stats: ops{read=9880 overwritten=0 applied=9877 ignored=0} inserts{seen=493800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20240503 08:11:36.203871 25281 consensus_peers.cc:485] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 -> Peer c5c48e7baba24a348bb082bf89fb4f29 (127.16.212.195:40417): Couldn't send request to peer c5c48e7baba24a348bb082bf89fb4f29. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:36.235085 25416 raft_consensus.cc:1261] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 4 FOLLOWER]: Refusing update from remote peer 705113ccef9e44d7a0913e9f532a1084: Log matching property violated. Preceding OpId in replica: term: 3 index: 13822. Preceding OpId from leader: term: 4 index: 14643. (index mismatch)
I20240503 08:11:36.236061 25562 consensus_queue.cc:1035] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14643, Last known committed idx: 13822, Time since last communication: 0.000s
I20240503 08:11:36.552134 25481 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Bootstrap replayed 3/3 log segments. Stats: ops{read=14642 overwritten=0 applied=14642 ignored=0} inserts{seen=671600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:11:36.552570 25481 tablet_bootstrap.cc:492] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Bootstrap complete.
I20240503 08:11:36.557190 25481 ts_tablet_manager.cc:1367] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Time spent bootstrapping tablet: real 2.375s user 2.023s sys 0.348s
I20240503 08:11:36.557556 25481 raft_consensus.cc:348] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:36.557678 25481 raft_consensus.cc:729] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c48e7baba24a348bb082bf89fb4f29, State: Initialized, Role: FOLLOWER
I20240503 08:11:36.557735 25481 consensus_queue.cc:260] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14642, Last appended: 3.14642, Last appended by leader: 14642, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5c48e7baba24a348bb082bf89fb4f29" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 40417 } } peers { permanent_uuid: "705113ccef9e44d7a0913e9f532a1084" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40849 } } peers { permanent_uuid: "bfc1ed9303764aada8541d8bbc31ed09" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 43021 } }
I20240503 08:11:36.557948 25481 ts_tablet_manager.cc:1398] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
W20240503 08:11:36.588510 25504 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51840: Illegal state: replica c5c48e7baba24a348bb082bf89fb4f29 is not leader of this config: current role FOLLOWER
I20240503 08:11:36.680910 25514 raft_consensus.cc:3037] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:11:37.061844 17235 tablet_server.cc:178] TabletServer@127.16.212.193:40849 shutting down...
I20240503 08:11:37.064491 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:37.064637 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:11:37.064703 17235 raft_consensus.cc:2229] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 4 LEADER]: Raft consensus shutting down.
I20240503 08:11:37.064769 17235 raft_consensus.cc:2258] T df9b5dd5e29948169ece9901d6742584 P 705113ccef9e44d7a0913e9f532a1084 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:37.074035 17235 tablet_server.cc:194] TabletServer@127.16.212.193:40849 shutdown complete.
I20240503 08:11:37.075451 17235 tablet_server.cc:178] TabletServer@127.16.212.194:43021 shutting down...
I20240503 08:11:37.078572 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:37.078691 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:11:37.078744 17235 raft_consensus.cc:2229] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:37.078800 17235 raft_consensus.cc:2258] T df9b5dd5e29948169ece9901d6742584 P bfc1ed9303764aada8541d8bbc31ed09 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:37.093873 17235 tablet_server.cc:194] TabletServer@127.16.212.194:43021 shutdown complete.
I20240503 08:11:37.096009 17235 tablet_server.cc:178] TabletServer@127.16.212.195:40417 shutting down...
I20240503 08:11:37.098656 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:37.098835 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:11:37.098906 17235 raft_consensus.cc:2229] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:37.098986 17235 raft_consensus.cc:2258] T df9b5dd5e29948169ece9901d6742584 P c5c48e7baba24a348bb082bf89fb4f29 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:37.125458 17235 tablet_server.cc:194] TabletServer@127.16.212.195:40417 shutdown complete.
I20240503 08:11:37.126482 17235 master.cc:537] Master@127.16.212.254:43847 shutting down...
I20240503 08:11:37.129254 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:37.129334 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:37.129367 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50201e51cbaf4f6592a2a57fa3f37fa2: stopping tablet replica
I20240503 08:11:37.141212 17235 master.cc:552] Master@127.16.212.254:43847 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap (19618 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound
I20240503 08:11:37.161914 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:40033
I20240503 08:11:37.162098 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.163941 25580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.164472 25579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.165171 17235 server_base.cc:1034] running on GCE node
W20240503 08:11:37.165894 25582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.166039 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:37.166083 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:37.166102 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723897166102 us; error 0 us; skew 500 ppm
I20240503 08:11:37.166806 17235 webserver.cc:460] Webserver started at http://127.16.212.254:37325/ using document root <none> and password file <none>
I20240503 08:11:37.166893 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.166937 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.166987 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:37.167256 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "87f54f973d444a29ab1b3cf1239b60ac"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.168115 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:37.168563 25587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.168694 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.168737 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "87f54f973d444a29ab1b3cf1239b60ac"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.168782 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.172735 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.172886 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.175302 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:40033
I20240503 08:11:37.179427 25639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:40033 every 8 connection(s)
I20240503 08:11:37.179545 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:37.180490 25640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac: Bootstrap starting.
I20240503 08:11:37.180763 25640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:37.181249 25640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac: No bootstrap required, opened a new log
I20240503 08:11:37.181363 25640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f54f973d444a29ab1b3cf1239b60ac" member_type: VOTER }
I20240503 08:11:37.181413 25640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:37.181437 25640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f54f973d444a29ab1b3cf1239b60ac, State: Initialized, Role: FOLLOWER
I20240503 08:11:37.181499 25640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f54f973d444a29ab1b3cf1239b60ac" member_type: VOTER }
I20240503 08:11:37.181533 25640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:37.181556 25640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:37.181587 25640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:37.182060 25640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f54f973d444a29ab1b3cf1239b60ac" member_type: VOTER }
I20240503 08:11:37.182114 25640 leader_election.cc:302] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f54f973d444a29ab1b3cf1239b60ac; no voters:
I20240503 08:11:37.182216 25640 leader_election.cc:288] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:37.182263 25643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:37.182349 25640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:37.182423 25643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 1 LEADER]: Becoming Leader. State: Replica: 87f54f973d444a29ab1b3cf1239b60ac, State: Running, Role: LEADER
I20240503 08:11:37.182488 25643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f54f973d444a29ab1b3cf1239b60ac" member_type: VOTER }
I20240503 08:11:37.182708 25644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f54f973d444a29ab1b3cf1239b60ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f54f973d444a29ab1b3cf1239b60ac" member_type: VOTER } }
I20240503 08:11:37.182873 25644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:37.182708 25645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f54f973d444a29ab1b3cf1239b60ac. Latest consensus state: current_term: 1 leader_uuid: "87f54f973d444a29ab1b3cf1239b60ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f54f973d444a29ab1b3cf1239b60ac" member_type: VOTER } }
I20240503 08:11:37.183017 25645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:37.183097 25651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:37.183233 25651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:37.183694 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:37.184077 25651 catalog_manager.cc:1347] Generated new cluster ID: c5120fee11d04c51af65d36e59652735
I20240503 08:11:37.184123 25651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:37.188575 25651 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:37.188982 25651 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:37.196072 25651 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac: Generated new TSK 0
I20240503 08:11:37.196146 25651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:37.199308 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.201365 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.202075 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.202651 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.203336 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:37.203472 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:37.203557 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:37.203594 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723897203594 us; error 0 us; skew 500 ppm
I20240503 08:11:37.204344 17235 webserver.cc:460] Webserver started at http://127.16.212.193:33955/ using document root <none> and password file <none>
I20240503 08:11:37.204428 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.204480 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.204530 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:37.204810 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "5e21f5fcd3764141bb6c19b46ceaacac"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.205771 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:37.206312 25669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.206457 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.206501 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "5e21f5fcd3764141bb6c19b46ceaacac"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.206563 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.216919 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.217096 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.217239 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.217419 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.217445 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.217478 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.217505 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.220217 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:37957
I20240503 08:11:37.220760 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:37.221118 25732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:37957 every 8 connection(s)
W20240503 08:11:37.222637 25736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.223166 25735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.223723 25738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.224627 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:37.224649 25733 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40033
I20240503 08:11:37.224712 25733 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.224726 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:37.224758 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:37.224776 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723897224776 us; error 0 us; skew 500 ppm
I20240503 08:11:37.224848 25733 heartbeater.cc:511] Master 127.16.212.254:40033 requested a full tablet report, sending...
I20240503 08:11:37.225085 25604 ts_manager.cc:194] Registered new tserver with Master: 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957)
I20240503 08:11:37.225497 17235 webserver.cc:460] Webserver started at http://127.16.212.194:41299/ using document root <none> and password file <none>
I20240503 08:11:37.225601 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.225649 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.225662 25604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58184
I20240503 08:11:37.225701 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:37.225968 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "2b647c866f964e27bcf857c65abf4a9d"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.227038 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:37.227562 25745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.227674 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:11:37.227733 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "2b647c866f964e27bcf857c65abf4a9d"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.227808 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.254144 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.254359 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.254535 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.254760 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.254789 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.254824 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.254855 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.257647 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:42837
I20240503 08:11:37.257688 25808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:42837 every 8 connection(s)
I20240503 08:11:37.258036 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:37.259970 25812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.260751 25813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:37.261250 25815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:37.261427 25809 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40033
I20240503 08:11:37.261497 25809 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.261610 25809 heartbeater.cc:511] Master 127.16.212.254:40033 requested a full tablet report, sending...
I20240503 08:11:37.261842 25604 ts_manager.cc:194] Registered new tserver with Master: 2b647c866f964e27bcf857c65abf4a9d (127.16.212.194:42837)
I20240503 08:11:37.261899 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:37.262014 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:37.262038 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:37.262056 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723897262056 us; error 0 us; skew 500 ppm
I20240503 08:11:37.262351 25604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58192
I20240503 08:11:37.262893 17235 webserver.cc:460] Webserver started at http://127.16.212.195:36185/ using document root <none> and password file <none>
I20240503 08:11:37.262987 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:37.263037 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:37.263074 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:37.263357 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "95d96a992242431cb221a9a12603dfec"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.264248 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:37.264688 25820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.264824 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.264869 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "95d96a992242431cb221a9a12603dfec"
format_stamp: "Formatted at 2024-05-03 08:11:37 on dist-test-slave-zl2t"
I20240503 08:11:37.264921 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:37.285566 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:37.285799 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:37.285964 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:37.286171 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:37.286201 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.286235 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:37.286262 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.289196 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:36165
I20240503 08:11:37.289245 25883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:36165 every 8 connection(s)
I20240503 08:11:37.292280 25884 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40033
I20240503 08:11:37.292335 25884 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:37.292433 25884 heartbeater.cc:511] Master 127.16.212.254:40033 requested a full tablet report, sending...
I20240503 08:11:37.292671 25604 ts_manager.cc:194] Registered new tserver with Master: 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165)
I20240503 08:11:37.292733 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003329614s
I20240503 08:11:37.293130 17235 test_util.cc:268] Using random seed: 1154038090
I20240503 08:11:37.293161 25604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58198
I20240503 08:11:37.297955 25604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58204:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:11:37.298096 25604 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:37.302707 25697 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec00dbd15844e3fae8825c3a15bf504 (DEFAULT_TABLE table=test-workload [id=fc63641d16924f67a9641573c3756b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:37.302832 25773 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec00dbd15844e3fae8825c3a15bf504 (DEFAULT_TABLE table=test-workload [id=fc63641d16924f67a9641573c3756b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:37.302848 25697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec00dbd15844e3fae8825c3a15bf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:37.302909 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec00dbd15844e3fae8825c3a15bf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:37.304284 25899 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d: Bootstrap starting.
I20240503 08:11:37.304342 25848 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec00dbd15844e3fae8825c3a15bf504 (DEFAULT_TABLE table=test-workload [id=fc63641d16924f67a9641573c3756b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:37.304455 25848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec00dbd15844e3fae8825c3a15bf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:37.304513 25900 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Bootstrap starting.
I20240503 08:11:37.304746 25899 tablet_bootstrap.cc:654] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:37.304975 25900 tablet_bootstrap.cc:654] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:37.305341 25899 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d: No bootstrap required, opened a new log
I20240503 08:11:37.305403 25899 ts_tablet_manager.cc:1367] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:37.305625 25899 raft_consensus.cc:348] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.305714 25900 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: No bootstrap required, opened a new log
I20240503 08:11:37.305740 25899 raft_consensus.cc:374] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:37.305766 25900 ts_tablet_manager.cc:1367] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:37.305820 25903 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: Bootstrap starting.
I20240503 08:11:37.305920 25900 raft_consensus.cc:348] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.305964 25900 raft_consensus.cc:374] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:37.305985 25900 raft_consensus.cc:729] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e21f5fcd3764141bb6c19b46ceaacac, State: Initialized, Role: FOLLOWER
I20240503 08:11:37.306035 25900 consensus_queue.cc:260] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.306180 25903 tablet_bootstrap.cc:654] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:37.306195 25900 ts_tablet_manager.cc:1398] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:37.306216 25733 heartbeater.cc:503] Master 127.16.212.254:40033 was elected leader, sending a full tablet report...
I20240503 08:11:37.305765 25899 raft_consensus.cc:729] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b647c866f964e27bcf857c65abf4a9d, State: Initialized, Role: FOLLOWER
I20240503 08:11:37.306303 25899 consensus_queue.cc:260] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.306488 25899 ts_tablet_manager.cc:1398] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:37.306579 25809 heartbeater.cc:503] Master 127.16.212.254:40033 was elected leader, sending a full tablet report...
I20240503 08:11:37.306849 25903 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: No bootstrap required, opened a new log
I20240503 08:11:37.306903 25903 ts_tablet_manager.cc:1367] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:37.307044 25903 raft_consensus.cc:348] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.307109 25903 raft_consensus.cc:374] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:37.307130 25903 raft_consensus.cc:729] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d96a992242431cb221a9a12603dfec, State: Initialized, Role: FOLLOWER
I20240503 08:11:37.307219 25903 consensus_queue.cc:260] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.307390 25903 ts_tablet_manager.cc:1398] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:37.307421 25884 heartbeater.cc:503] Master 127.16.212.254:40033 was elected leader, sending a full tablet report...
I20240503 08:11:37.340406 25904 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:37.340509 25904 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.340695 25904 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165), 2b647c866f964e27bcf857c65abf4a9d (127.16.212.194:42837)
I20240503 08:11:37.343986 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b647c866f964e27bcf857c65abf4a9d" is_pre_election: true
I20240503 08:11:37.343986 25858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d96a992242431cb221a9a12603dfec" is_pre_election: true
I20240503 08:11:37.344147 25783 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e21f5fcd3764141bb6c19b46ceaacac in term 0.
I20240503 08:11:37.344178 25858 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e21f5fcd3764141bb6c19b46ceaacac in term 0.
I20240503 08:11:37.344316 25672 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b647c866f964e27bcf857c65abf4a9d, 5e21f5fcd3764141bb6c19b46ceaacac; no voters:
I20240503 08:11:37.344448 25904 raft_consensus.cc:2781] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:37.344499 25904 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:37.344519 25904 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:37.345197 25904 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.345397 25904 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [CANDIDATE]: Term 1 election: Requested vote from peers 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165), 2b647c866f964e27bcf857c65abf4a9d (127.16.212.194:42837)
I20240503 08:11:37.345472 25858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d96a992242431cb221a9a12603dfec"
I20240503 08:11:37.345496 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b647c866f964e27bcf857c65abf4a9d"
I20240503 08:11:37.345537 25858 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:37.345567 25783 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:37.346119 25858 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e21f5fcd3764141bb6c19b46ceaacac in term 1.
I20240503 08:11:37.346181 25783 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e21f5fcd3764141bb6c19b46ceaacac in term 1.
I20240503 08:11:37.346300 25671 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e21f5fcd3764141bb6c19b46ceaacac, 95d96a992242431cb221a9a12603dfec; no voters:
I20240503 08:11:37.346405 25904 raft_consensus.cc:2781] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:37.346561 25904 raft_consensus.cc:686] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 LEADER]: Becoming Leader. State: Replica: 5e21f5fcd3764141bb6c19b46ceaacac, State: Running, Role: LEADER
I20240503 08:11:37.346634 25904 consensus_queue.cc:237] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:37.347291 25604 catalog_manager.cc:5484] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193). New cstate: current_term: 1 leader_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:37.368140 25858 raft_consensus.cc:1261] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Refusing update from remote peer 5e21f5fcd3764141bb6c19b46ceaacac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:37.368140 25783 raft_consensus.cc:1261] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Refusing update from remote peer 5e21f5fcd3764141bb6c19b46ceaacac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:37.368423 25909 consensus_queue.cc:1035] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [LEADER]: Connected to new peer: Peer: permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:37.368539 25904 consensus_queue.cc:1035] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:37.370391 25916 mvcc.cc:205] Tried to move back new op lower bound from 7023509083620421632 to 7023509083532386304. Current Snapshot: MvccSnapshot[applied={T|T < 7023509083618590720}]
I20240503 08:11:47.367561 17235 tablet_replica.cc:330] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: stopping tablet replica
I20240503 08:11:47.367637 17235 raft_consensus.cc:2229] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:47.367702 17235 raft_consensus.cc:2258] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:47.369535 17235 ts_tablet_manager.cc:1875] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 08:11:47.371359 17235 ts_tablet_manager.cc:1888] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13439
I20240503 08:11:47.371428 17235 log.cc:1196] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Deleting WAL directory at /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/8ec00dbd15844e3fae8825c3a15bf504
I20240503 08:11:48.860671 25964 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e21f5fcd3764141bb6c19b46ceaacac)
I20240503 08:11:48.860818 25965 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e21f5fcd3764141bb6c19b46ceaacac)
I20240503 08:11:48.860845 25964 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:48.860992 25965 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:48.861177 25964 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957), 2b647c866f964e27bcf857c65abf4a9d (127.16.212.194:42837)
I20240503 08:11:48.861193 25965 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165), 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957)
I20240503 08:11:48.864676 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "95d96a992242431cb221a9a12603dfec" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "2b647c866f964e27bcf857c65abf4a9d" is_pre_election: true
I20240503 08:11:48.864763 25783 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d96a992242431cb221a9a12603dfec in term 1.
I20240503 08:11:48.864953 25858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "2b647c866f964e27bcf857c65abf4a9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "95d96a992242431cb221a9a12603dfec" is_pre_election: true
I20240503 08:11:48.864989 25823 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b647c866f964e27bcf857c65abf4a9d, 95d96a992242431cb221a9a12603dfec; no voters:
I20240503 08:11:48.865006 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "2b647c866f964e27bcf857c65abf4a9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" is_pre_election: true
I20240503 08:11:48.865041 25858 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b647c866f964e27bcf857c65abf4a9d in term 1.
I20240503 08:11:48.865103 25707 raft_consensus.cc:1789] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.13439
I20240503 08:11:48.865139 25707 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b647c866f964e27bcf857c65abf4a9d in term 1.
I20240503 08:11:48.865203 25964 raft_consensus.cc:2781] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:48.865255 25964 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e21f5fcd3764141bb6c19b46ceaacac)
I20240503 08:11:48.865275 25964 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:48.865214 25747 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b647c866f964e27bcf857c65abf4a9d, 95d96a992242431cb221a9a12603dfec; no voters:
I20240503 08:11:48.865443 25965 raft_consensus.cc:2781] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 08:11:48.865494 25965 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e21f5fcd3764141bb6c19b46ceaacac)
I20240503 08:11:48.865520 25965 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:48.865900 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "95d96a992242431cb221a9a12603dfec" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" is_pre_election: true
I20240503 08:11:48.865967 25707 raft_consensus.cc:1789] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.13439
I20240503 08:11:48.865998 25707 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d96a992242431cb221a9a12603dfec in term 1.
I20240503 08:11:48.866225 25964 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:48.866341 25964 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [CANDIDATE]: Term 2 election: Requested vote from peers 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957), 2b647c866f964e27bcf857c65abf4a9d (127.16.212.194:42837)
I20240503 08:11:48.866389 25965 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:48.866478 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "95d96a992242431cb221a9a12603dfec" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "5e21f5fcd3764141bb6c19b46ceaacac"
I20240503 08:11:48.866547 25707 raft_consensus.cc:1789] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.13439
I20240503 08:11:48.866590 25707 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:11:48.866524 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "95d96a992242431cb221a9a12603dfec" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "2b647c866f964e27bcf857c65abf4a9d"
I20240503 08:11:48.866683 25783 raft_consensus.cc:2379] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 95d96a992242431cb221a9a12603dfec in current term 2: Already voted for candidate 2b647c866f964e27bcf857c65abf4a9d in this term.
I20240503 08:11:48.866688 25858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "2b647c866f964e27bcf857c65abf4a9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "95d96a992242431cb221a9a12603dfec"
I20240503 08:11:48.866691 25965 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [CANDIDATE]: Term 2 election: Requested vote from peers 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165), 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957)
I20240503 08:11:48.866784 25858 raft_consensus.cc:2379] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b647c866f964e27bcf857c65abf4a9d in current term 2: Already voted for candidate 95d96a992242431cb221a9a12603dfec in this term.
I20240503 08:11:48.867255 25707 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d96a992242431cb221a9a12603dfec in term 2.
I20240503 08:11:48.867373 25706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "2b647c866f964e27bcf857c65abf4a9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 13439 } } ignore_live_leader: false dest_uuid: "5e21f5fcd3764141bb6c19b46ceaacac"
I20240503 08:11:48.867398 25823 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e21f5fcd3764141bb6c19b46ceaacac, 95d96a992242431cb221a9a12603dfec; no voters: 2b647c866f964e27bcf857c65abf4a9d
I20240503 08:11:48.867424 25706 raft_consensus.cc:1789] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.13439
I20240503 08:11:48.867447 25706 raft_consensus.cc:2379] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b647c866f964e27bcf857c65abf4a9d in current term 2: Already voted for candidate 95d96a992242431cb221a9a12603dfec in this term.
I20240503 08:11:48.867527 25964 raft_consensus.cc:2781] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:11:48.867610 25748 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b647c866f964e27bcf857c65abf4a9d; no voters: 5e21f5fcd3764141bb6c19b46ceaacac, 95d96a992242431cb221a9a12603dfec
I20240503 08:11:48.867679 25964 raft_consensus.cc:686] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 LEADER]: Becoming Leader. State: Replica: 95d96a992242431cb221a9a12603dfec, State: Running, Role: LEADER
I20240503 08:11:48.867735 25964 consensus_queue.cc:237] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13439, Committed index: 13439, Last appended: 1.13439, Last appended by leader: 13439, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:48.867974 25965 raft_consensus.cc:2726] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:11:48.868216 25598 catalog_manager.cc:5484] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec reported cstate change: term changed from 1 to 2, leader changed from 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193) to 95d96a992242431cb221a9a12603dfec (127.16.212.195). New cstate: current_term: 2 leader_uuid: "95d96a992242431cb221a9a12603dfec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:48.926285 25783 raft_consensus.cc:1261] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 2 FOLLOWER]: Refusing update from remote peer 95d96a992242431cb221a9a12603dfec: Log matching property violated. Preceding OpId in replica: term: 1 index: 13439. Preceding OpId from leader: term: 2 index: 13441. (index mismatch)
W20240503 08:11:48.926430 25823 consensus_peers.cc:485] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec -> Peer 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957): Couldn't send request to peer 5e21f5fcd3764141bb6c19b46ceaacac. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:11:48.926578 25974 consensus_queue.cc:1035] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13440, Last known committed idx: 13439, Time since last communication: 0.000s
I20240503 08:11:49.321233 25988 ts_tablet_manager.cc:902] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Initiating tablet copy from peer 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165)
I20240503 08:11:49.321661 25988 tablet_copy_client.cc:308] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: tablet copy: Beginning tablet copy session from remote peer at address 127.16.212.195:36165
I20240503 08:11:49.322010 25868 tablet_copy_service.cc:140] P 95d96a992242431cb221a9a12603dfec: Received BeginTabletCopySession request for tablet 8ec00dbd15844e3fae8825c3a15bf504 from peer 5e21f5fcd3764141bb6c19b46ceaacac ({username='slave'} at 127.0.0.1:42710)
I20240503 08:11:49.322082 25868 tablet_copy_service.cc:161] P 95d96a992242431cb221a9a12603dfec: Beginning new tablet copy session on tablet 8ec00dbd15844e3fae8825c3a15bf504 from peer 5e21f5fcd3764141bb6c19b46ceaacac at {username='slave'} at 127.0.0.1:42710: session id = 5e21f5fcd3764141bb6c19b46ceaacac-8ec00dbd15844e3fae8825c3a15bf504
I20240503 08:11:49.322566 25868 tablet_copy_source_session.cc:205] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: Tablet Copy: opened 0 blocks and 3 log segments
I20240503 08:11:49.323019 25988 ts_tablet_manager.cc:1875] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 08:11:49.324576 25988 ts_tablet_manager.cc:1888] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.13439
I20240503 08:11:49.324667 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec00dbd15844e3fae8825c3a15bf504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:49.325204 25988 tablet_copy_client.cc:783] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: tablet copy: Starting download of 0 data blocks...
I20240503 08:11:49.325374 25988 tablet_copy_client.cc:647] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: tablet copy: Starting download of 3 WAL segments...
I20240503 08:11:49.363124 25988 tablet_copy_client.cc:523] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:11:49.363948 25988 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Bootstrap starting.
I20240503 08:11:50.202365 25988 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Bootstrap replayed 1/3 log segments. Stats: ops{read=4942 overwritten=0 applied=4941 ignored=0} inserts{seen=247000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:51.045352 25988 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Bootstrap replayed 2/3 log segments. Stats: ops{read=9875 overwritten=0 applied=9873 ignored=0} inserts{seen=493600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
W20240503 08:11:51.673358 25823 consensus_peers.cc:485] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec -> Peer 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193:37957): Couldn't send request to peer 5e21f5fcd3764141bb6c19b46ceaacac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 08:11:51.680986 25988 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Bootstrap replayed 3/3 log segments. Stats: ops{read=13870 overwritten=0 applied=13869 ignored=0} inserts{seen=671900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:11:51.681258 25988 tablet_bootstrap.cc:492] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Bootstrap complete.
I20240503 08:11:51.685889 25988 ts_tablet_manager.cc:1367] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Time spent bootstrapping tablet: real 2.322s user 1.995s sys 0.324s
I20240503 08:11:51.686056 25988 raft_consensus.cc:348] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:51.686232 25988 raft_consensus.cc:729] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e21f5fcd3764141bb6c19b46ceaacac, State: Initialized, Role: FOLLOWER
I20240503 08:11:51.686296 25988 consensus_queue.cc:260] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13869, Last appended: 2.13870, Last appended by leader: 13870, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:51.686420 25988 ts_tablet_manager.cc:1398] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Time spent starting tablet: real 0.000s user 0.003s sys 0.001s
I20240503 08:11:51.686745 25866 tablet_copy_service.cc:342] P 95d96a992242431cb221a9a12603dfec: Request end of tablet copy session 5e21f5fcd3764141bb6c19b46ceaacac-8ec00dbd15844e3fae8825c3a15bf504 received from {username='slave'} at 127.0.0.1:42710
I20240503 08:11:51.686846 25866 tablet_copy_service.cc:433] P 95d96a992242431cb221a9a12603dfec: ending tablet copy session 5e21f5fcd3764141bb6c19b46ceaacac-8ec00dbd15844e3fae8825c3a15bf504 on tablet 8ec00dbd15844e3fae8825c3a15bf504 with peer 5e21f5fcd3764141bb6c19b46ceaacac
I20240503 08:11:51.709014 25974 raft_consensus.cc:979] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: : Instructing follower 2b647c866f964e27bcf857c65abf4a9d to start an election
I20240503 08:11:51.709110 25974 raft_consensus.cc:1067] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 LEADER]: Signalling peer 2b647c866f964e27bcf857c65abf4a9d to start an election
I20240503 08:11:51.709296 25783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504"
dest_uuid: "2b647c866f964e27bcf857c65abf4a9d"
from {username='slave'} at 127.0.0.1:33248
W20240503 08:11:51.709486 25823 consensus_peers.cc:330] unable to start election on peer 2b647c866f964e27bcf857c65abf4a9d: Illegal state: leader elections are disabled
I20240503 08:11:52.106272 25706 raft_consensus.cc:1203] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.13439->[2.13440-2.14082] Dedup: 2.13870->[2.13871-2.14082]
I20240503 08:11:52.109416 25987 raft_consensus.cc:979] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: : Instructing follower 5e21f5fcd3764141bb6c19b46ceaacac to start an election
I20240503 08:11:52.109505 25987 raft_consensus.cc:1067] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 LEADER]: Signalling peer 5e21f5fcd3764141bb6c19b46ceaacac to start an election
I20240503 08:11:52.109781 25706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504"
dest_uuid: "5e21f5fcd3764141bb6c19b46ceaacac"
from {username='slave'} at 127.0.0.1:51228
I20240503 08:11:52.109869 25706 raft_consensus.cc:482] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:11:52.109900 25706 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:52.110792 25706 raft_consensus.cc:504] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:52.111129 25706 leader_election.cc:288] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [CANDIDATE]: Term 3 election: Requested vote from peers 95d96a992242431cb221a9a12603dfec (127.16.212.195:36165), 2b647c866f964e27bcf857c65abf4a9d (127.16.212.194:42837)
I20240503 08:11:52.111191 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" candidate_term: 3 candidate_status { last_received { term: 2 index: 14082 } } ignore_live_leader: true dest_uuid: "2b647c866f964e27bcf857c65abf4a9d"
I20240503 08:11:52.111168 25858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec00dbd15844e3fae8825c3a15bf504" candidate_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" candidate_term: 3 candidate_status { last_received { term: 2 index: 14082 } } ignore_live_leader: true dest_uuid: "95d96a992242431cb221a9a12603dfec"
I20240503 08:11:52.111256 25783 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:52.111266 25858 raft_consensus.cc:3032] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 LEADER]: Stepping down as leader of term 2
I20240503 08:11:52.111299 25858 raft_consensus.cc:729] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95d96a992242431cb221a9a12603dfec, State: Running, Role: LEADER
I20240503 08:11:52.111346 25858 consensus_queue.cc:260] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14082, Committed index: 14082, Last appended: 2.14082, Last appended by leader: 14082, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:52.111454 25858 raft_consensus.cc:3037] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:11:52.111835 25783 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e21f5fcd3764141bb6c19b46ceaacac in term 3.
I20240503 08:11:52.112098 25672 leader_election.cc:302] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b647c866f964e27bcf857c65abf4a9d, 5e21f5fcd3764141bb6c19b46ceaacac; no voters:
I20240503 08:11:52.112154 25858 raft_consensus.cc:2454] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e21f5fcd3764141bb6c19b46ceaacac in term 3.
I20240503 08:11:52.112236 25997 raft_consensus.cc:2781] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:11:52.112329 25997 raft_consensus.cc:686] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 3 LEADER]: Becoming Leader. State: Replica: 5e21f5fcd3764141bb6c19b46ceaacac, State: Running, Role: LEADER
I20240503 08:11:52.112386 25997 consensus_queue.cc:237] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14082, Committed index: 14082, Last appended: 2.14082, Last appended by leader: 14082, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } }
I20240503 08:11:52.112792 25598 catalog_manager.cc:5484] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac reported cstate change: term changed from 2 to 3, leader changed from 95d96a992242431cb221a9a12603dfec (127.16.212.195) to 5e21f5fcd3764141bb6c19b46ceaacac (127.16.212.193). New cstate: current_term: 3 leader_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d96a992242431cb221a9a12603dfec" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 36165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e21f5fcd3764141bb6c19b46ceaacac" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b647c866f964e27bcf857c65abf4a9d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42837 } health_report { overall_health: UNKNOWN } } }
I20240503 08:11:52.225245 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:11:52.228137 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.228300 17235 tablet_replica.cc:330] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: stopping tablet replica
I20240503 08:11:52.228371 17235 raft_consensus.cc:2229] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 3 LEADER]: Raft consensus shutting down.
I20240503 08:11:52.228554 17235 pending_rounds.cc:62] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Trying to abort 1 pending ops.
I20240503 08:11:52.228577 17235 pending_rounds.cc:68] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac: Aborting op as it isn't in flight: id { term: 3 index: 14083 } timestamp: 7023509144012853248 op_type: NO_OP noop_request { }
I20240503 08:11:52.228636 17235 raft_consensus.cc:2866] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 08:11:52.228667 17235 raft_consensus.cc:2258] T 8ec00dbd15844e3fae8825c3a15bf504 P 5e21f5fcd3764141bb6c19b46ceaacac [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.247402 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:11:52.248570 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:11:52.251935 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.252059 17235 tablet_replica.cc:330] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d: stopping tablet replica
I20240503 08:11:52.252117 17235 raft_consensus.cc:2229] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:52.252915 17235 raft_consensus.cc:2258] T 8ec00dbd15844e3fae8825c3a15bf504 P 2b647c866f964e27bcf857c65abf4a9d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.267498 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:11:52.268694 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:11:52.271358 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:11:52.271456 17235 tablet_replica.cc:330] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec: stopping tablet replica
I20240503 08:11:52.271517 17235 raft_consensus.cc:2229] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 08:11:52.272028 17235 raft_consensus.cc:2258] T 8ec00dbd15844e3fae8825c3a15bf504 P 95d96a992242431cb221a9a12603dfec [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.297559 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:11:52.298744 17235 master.cc:537] Master@127.16.212.254:40033 shutting down...
I20240503 08:11:52.300873 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:11:52.300942 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:11:52.300962 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 87f54f973d444a29ab1b3cf1239b60ac: stopping tablet replica
I20240503 08:11:52.312929 17235 master.cc:552] Master@127.16.212.254:40033 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound (15170 ms)
[----------] 21 tests from TxnParticipantITest (252361 ms total)
[----------] 2 tests from TxnParticipantElectionStormITest
[ RUN ] TxnParticipantElectionStormITest.TestFrequentElections
I20240503 08:11:52.332024 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:40039
I20240503 08:11:52.332175 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:52.334472 17235 server_base.cc:1034] running on GCE node
W20240503 08:11:52.335289 26006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.335886 26007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.336338 26009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.336501 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:52.336544 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:52.336567 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723912336567 us; error 0 us; skew 500 ppm
I20240503 08:11:52.342132 17235 webserver.cc:460] Webserver started at http://127.16.212.254:40767/ using document root <none> and password file <none>
I20240503 08:11:52.342257 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:52.342401 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:52.342476 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:52.342851 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "b3cc48360c95405c90eb4737e611be95"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.343791 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:52.344321 26014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.344507 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.344563 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "b3cc48360c95405c90eb4737e611be95"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.344616 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:52.354969 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:52.355178 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:52.357743 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:40039
I20240503 08:11:52.359696 26066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:40039 every 8 connection(s)
I20240503 08:11:52.364110 26067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:52.365137 26067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Bootstrap starting.
I20240503 08:11:52.365384 26067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:52.365895 26067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: No bootstrap required, opened a new log
I20240503 08:11:52.366000 26067 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:11:52.366067 26067 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:52.366087 26067 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc48360c95405c90eb4737e611be95, State: Initialized, Role: FOLLOWER
I20240503 08:11:52.366142 26067 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:11:52.366184 26067 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:11:52.366204 26067 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:11:52.366230 26067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.366729 26067 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:11:52.366783 26067 leader_election.cc:302] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3cc48360c95405c90eb4737e611be95; no voters:
I20240503 08:11:52.366865 26067 leader_election.cc:288] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:11:52.366923 26070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:52.367007 26067 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:11:52.367072 26070 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 LEADER]: Becoming Leader. State: Replica: b3cc48360c95405c90eb4737e611be95, State: Running, Role: LEADER
I20240503 08:11:52.367128 26070 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:11:52.367431 26071 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3cc48360c95405c90eb4737e611be95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER } }
I20240503 08:11:52.367455 26072 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3cc48360c95405c90eb4737e611be95. Latest consensus state: current_term: 1 leader_uuid: "b3cc48360c95405c90eb4737e611be95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER } }
I20240503 08:11:52.367517 26071 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:52.367525 26072 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: This master's current role is: LEADER
I20240503 08:11:52.367677 26078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:11:52.367841 26078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:11:52.368428 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:11:52.368434 26078 catalog_manager.cc:1347] Generated new cluster ID: 4bb74111b3f64356abf9aba4b8e25d0d
I20240503 08:11:52.368505 26078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:11:52.392695 26078 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:11:52.393143 26078 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:11:52.398187 26078 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Generated new TSK 0
I20240503 08:11:52.398257 26078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:11:52.400208 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:11:52.402263 26089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.403062 26088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.403564 26091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.404081 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:52.404165 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:52.404192 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:52.404214 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723912404214 us; error 0 us; skew 500 ppm
I20240503 08:11:52.404946 17235 webserver.cc:460] Webserver started at http://127.16.212.193:45897/ using document root <none> and password file <none>
I20240503 08:11:52.405054 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:52.405102 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:52.405153 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:52.405432 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "db1737abb1d5409bb049fa1c94d3ed48"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.406322 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:52.406834 26096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.407001 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.407048 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "db1737abb1d5409bb049fa1c94d3ed48"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.407101 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:52.450973 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:52.451189 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:52.451367 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:52.451573 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:52.451603 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.451637 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:52.451655 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.455381 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:33927
I20240503 08:11:52.456051 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:52.456781 26159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:33927 every 8 connection(s)
W20240503 08:11:52.458500 26163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.459228 26162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.459826 26165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.460407 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:52.460511 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:52.460546 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:52.460563 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723912460563 us; error 0 us; skew 500 ppm
I20240503 08:11:52.460896 26160 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40039
I20240503 08:11:52.460954 26160 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:52.461066 26160 heartbeater.cc:511] Master 127.16.212.254:40039 requested a full tablet report, sending...
I20240503 08:11:52.461325 26031 ts_manager.cc:194] Registered new tserver with Master: db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927)
I20240503 08:11:52.461337 17235 webserver.cc:460] Webserver started at http://127.16.212.194:33407/ using document root <none> and password file <none>
I20240503 08:11:52.461438 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:52.461488 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:52.461540 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:52.461839 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.461970 26031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40572
I20240503 08:11:52.462891 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:52.463340 26172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.463486 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.463532 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.463569 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:52.473013 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:52.473224 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:52.473356 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:52.473532 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:52.473557 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.473579 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:52.473610 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.476277 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:40443
I20240503 08:11:52.476815 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:11:52.477569 26235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:40443 every 8 connection(s)
W20240503 08:11:52.479128 26239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.479724 26238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:11:52.480213 26241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:11:52.480985 17235 server_base.cc:1034] running on GCE node
I20240503 08:11:52.481019 26236 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40039
I20240503 08:11:52.481077 26236 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:52.481125 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:11:52.481161 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:11:52.481189 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723912481188 us; error 0 us; skew 500 ppm
I20240503 08:11:52.481197 26236 heartbeater.cc:511] Master 127.16.212.254:40039 requested a full tablet report, sending...
I20240503 08:11:52.481451 26031 ts_manager.cc:194] Registered new tserver with Master: 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:11:52.481889 26031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40578
I20240503 08:11:52.481937 17235 webserver.cc:460] Webserver started at http://127.16.212.195:46645/ using document root <none> and password file <none>
I20240503 08:11:52.482054 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:11:52.482103 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:11:52.482153 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:11:52.482439 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.483331 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:11:52.483744 26247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.483912 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.483965 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:11:52.483997 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:11:52.493465 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:11:52.493644 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:11:52.493780 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:11:52.493973 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:11:52.494005 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.494029 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:11:52.494055 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:11:52.497175 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:44771
I20240503 08:11:52.497289 26310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:44771 every 8 connection(s)
I20240503 08:11:52.500228 26311 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40039
I20240503 08:11:52.500283 26311 heartbeater.cc:464] Registering TS with master...
I20240503 08:11:52.500386 26311 heartbeater.cc:511] Master 127.16.212.254:40039 requested a full tablet report, sending...
I20240503 08:11:52.500670 26031 ts_manager.cc:194] Registered new tserver with Master: e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:11:52.501016 26031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40586
I20240503 08:11:52.501724 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004371696s
I20240503 08:11:52.502090 17235 test_util.cc:268] Using random seed: 1169247050
I20240503 08:11:52.506659 26031 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40590:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:11:52.506836 26031 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:11:52.511391 26200 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba72841b27b4d44b8f91e77a6d9f162 (DEFAULT_TABLE table=test-workload [id=eb268649e74041bd8a290edbd04ed5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:52.511523 26200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba72841b27b4d44b8f91e77a6d9f162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:52.511838 26124 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba72841b27b4d44b8f91e77a6d9f162 (DEFAULT_TABLE table=test-workload [id=eb268649e74041bd8a290edbd04ed5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:52.511909 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba72841b27b4d44b8f91e77a6d9f162 (DEFAULT_TABLE table=test-workload [id=eb268649e74041bd8a290edbd04ed5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:11:52.511968 26124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba72841b27b4d44b8f91e77a6d9f162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:52.511996 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba72841b27b4d44b8f91e77a6d9f162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:11:52.513407 26327 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Bootstrap starting.
I20240503 08:11:52.513535 26328 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Bootstrap starting.
I20240503 08:11:52.513849 26327 tablet_bootstrap.cc:654] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:52.513963 26328 tablet_bootstrap.cc:654] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:52.513386 26326 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap starting.
I20240503 08:11:52.514394 26327 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: No bootstrap required, opened a new log
I20240503 08:11:52.514433 26327 ts_tablet_manager.cc:1367] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:52.514546 26328 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: No bootstrap required, opened a new log
I20240503 08:11:52.514597 26328 ts_tablet_manager.cc:1367] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:52.514578 26327 raft_consensus.cc:348] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.514719 26327 raft_consensus.cc:374] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:52.514755 26327 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Initialized, Role: FOLLOWER
I20240503 08:11:52.514798 26328 raft_consensus.cc:348] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.514806 26327 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.514881 26328 raft_consensus.cc:374] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:52.514915 26328 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Initialized, Role: FOLLOWER
I20240503 08:11:52.514957 26326 tablet_bootstrap.cc:654] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Neither blocks nor log segments found. Creating new log.
I20240503 08:11:52.515007 26327 ts_tablet_manager.cc:1398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:11:52.514987 26328 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.515098 26160 heartbeater.cc:503] Master 127.16.212.254:40039 was elected leader, sending a full tablet report...
I20240503 08:11:52.515318 26328 ts_tablet_manager.cc:1398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 08:11:52.515339 26311 heartbeater.cc:503] Master 127.16.212.254:40039 was elected leader, sending a full tablet report...
I20240503 08:11:52.515697 26326 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: No bootstrap required, opened a new log
I20240503 08:11:52.515750 26326 ts_tablet_manager.cc:1367] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:11:52.515905 26326 raft_consensus.cc:348] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.515973 26326 raft_consensus.cc:374] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:11:52.515995 26326 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Initialized, Role: FOLLOWER
I20240503 08:11:52.516042 26326 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.516196 26326 ts_tablet_manager.cc:1398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:11:52.516294 26236 heartbeater.cc:503] Master 127.16.212.254:40039 was elected leader, sending a full tablet report...
I20240503 08:11:52.525249 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:52.525322 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.525475 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:11:52.526003 26331 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:52.526062 26331 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.526213 26331 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:11:52.526289 26333 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:11:52.526324 26333 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.526439 26333 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:11:52.528746 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48" is_pre_election: true
I20240503 08:11:52.528857 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 0.
I20240503 08:11:52.528973 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" is_pre_election: true
I20240503 08:11:52.528996 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:11:52.529036 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 0.
I20240503 08:11:52.529428 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:11:52.529515 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:11:52.530020 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.530215 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" is_pre_election: true
I20240503 08:11:52.531002 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.531248 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 1 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:11:52.531370 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:11:52.531447 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.531598 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" is_pre_election: true
I20240503 08:11:52.531627 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:11:52.531685 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:11:52.532217 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 1.
I20240503 08:11:52.532373 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 1.
I20240503 08:11:52.532389 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:11:52.531606 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 1: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:11:52.532610 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:11:52.532660 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48" is_pre_election: true
I20240503 08:11:52.532737 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 1: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:11:52.532747 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:11:52.532825 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:11:52.532847 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:11:52.532966 26331 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:52.533361 26031 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 1 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:11:52.533593 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" is_pre_election: true
I20240503 08:11:52.533872 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:11:52.533962 26333 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:11:52.541738 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 1 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:52.541754 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 1 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:11:52.541996 26341 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:11:52.542248 26341 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:02.569031 26398 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:02.569116 26398 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:02.569864 26398 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.570024 26398 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 2 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.570235 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 14233 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.570245 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 2 candidate_status { last_received { term: 1 index: 14233 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.570531 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:02.570683 26398 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:12:02.571897 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 1. Current term is 2. Ops: []
I20240503 08:12:02.572150 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 1.14233, Next index: 14234, Last known committed idx: 14233, Time since last communication: 0.000s
W20240503 08:12:02.572285 26384 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240503 08:12:02.572314 26384 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240503 08:12:02.597525 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:02.597636 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:02.598516 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.598932 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 2 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.599135 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 14233 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.599232 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 14233 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.599210 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 1. Current term is 2. Ops: [1.14234-1.14234]
I20240503 08:12:02.599211 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 2 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 1. Current term is 2. Ops: [1.14234-1.14234]
I20240503 08:12:02.599373 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 2: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:02.599509 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 1.14233, Next index: 14234, Last known committed idx: 14233, Time since last communication: 0.000s
I20240503 08:12:02.599551 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.599622 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 1.14233, Next index: 14234, Last known committed idx: 14233, Time since last communication: 0.000s
I20240503 08:12:02.599653 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:12:02.602430 26384 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:02.602483 26384 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:02.602555 26384 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14233, Committed index: 14233, Last appended: 1.14234, Last appended by leader: 14234, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.602669 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:02.610357 26398 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.610456 26398 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:02.611315 26398 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.611503 26398 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 3 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.611714 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 3 candidate_status { last_received { term: 1 index: 14233 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.611805 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:02.611717 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 3 candidate_status { last_received { term: 1 index: 14233 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.611852 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:02.612521 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 3.
I20240503 08:12:02.612641 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 3 because replica has last-logged OpId of term: 1 index: 14234, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 14233.
I20240503 08:12:02.612732 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:02.612820 26398 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:02.612942 26398 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 3 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:02.612998 26398 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14233, Committed index: 14233, Last appended: 1.14233, Last appended by leader: 14233, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.613479 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 1 to 3, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 3 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.624907 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.624991 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:02.625664 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.625818 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 4 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:02.626008 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 4 candidate_status { last_received { term: 1 index: 14234 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.626075 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:02.626066 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 4 candidate_status { last_received { term: 1 index: 14234 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.626777 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 4.
I20240503 08:12:02.626945 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.627027 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 FOLLOWER]: Leader election won for term 4
I20240503 08:12:02.627074 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:02.627133 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14233, Committed index: 14233, Last appended: 1.14234, Last appended by leader: 14234, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.639879 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 3. Current term is 4. Ops: []
I20240503 08:12:02.640161 26398 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14234, Last known committed idx: 14233, Time since last communication: 0.000s
I20240503 08:12:02.640275 26398 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 4.14235 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14233
I20240503 08:12:02.641498 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 4 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 3. Current term is 4. Ops: []
I20240503 08:12:02.641710 26398 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14234, Last known committed idx: 14233, Time since last communication: 0.000s
I20240503 08:12:02.645373 26407 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:02.645450 26407 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:02.645529 26407 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14233, Committed index: 14233, Last appended: 3.14234, Last appended by leader: 14234, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.645658 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 3 FOLLOWER]: Advancing to term 4
W20240503 08:12:02.653746 26384 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 4 is higher.
I20240503 08:12:02.654263 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 4 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 1 index: 14233. Preceding OpId from leader: term: 4 index: 14235. (index mismatch)
I20240503 08:12:02.654526 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14235, Last known committed idx: 14233, Time since last communication: 0.000s
I20240503 08:12:02.657629 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 3 to 4, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 4 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:02.660765 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 4 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 3 index: 14234. Preceding OpId from leader: term: 4 index: 14235. (index mismatch)
I20240503 08:12:02.660977 26390 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14235, Last known committed idx: 14233, Time since last communication: 0.000s
I20240503 08:12:02.661093 26390 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 3.14234 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14233
I20240503 08:12:02.661329 26134 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14233
I20240503 08:12:02.661386 26134 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 3.14234
I20240503 08:12:02.661408 26134 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240503 08:12:02.677028 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 4 is higher.
I20240503 08:12:02.686332 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:02.686431 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:02.687199 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.687361 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 5 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.687542 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 14235 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.687546 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 5 candidate_status { last_received { term: 4 index: 14235 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.687835 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.687937 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 08:12:02.692308 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 5 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 4. Current term is 5. Ops: []
I20240503 08:12:02.692541 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 4.14235, Next index: 14236, Last known committed idx: 14235, Time since last communication: 0.000s
I20240503 08:12:02.719055 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 5 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 4. Current term is 5. Ops: [4.14236-4.14236]
I20240503 08:12:02.719321 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 4.14235, Next index: 14236, Last known committed idx: 14235, Time since last communication: 0.000s
I20240503 08:12:02.722461 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.722527 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:12:02.723161 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.723297 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 6 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.723433 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 6 candidate_status { last_received { term: 4 index: 14235 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.723459 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 6 candidate_status { last_received { term: 4 index: 14235 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.723690 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.723788 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 08:12:02.749053 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 6 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 4. Current term is 6. Ops: [4.14236-4.14236]
I20240503 08:12:02.749308 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 4.14235, Next index: 14236, Last known committed idx: 14235, Time since last communication: 0.000s
I20240503 08:12:02.780277 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 6 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 4. Current term is 6. Ops: [4.14236-4.14236]
I20240503 08:12:02.780539 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 4.14235, Next index: 14236, Last known committed idx: 14235, Time since last communication: 0.000s
I20240503 08:12:02.784678 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.784776 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:12:02.785482 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.785653 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 7 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.785801 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 7 candidate_status { last_received { term: 4 index: 14235 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.785848 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 7 candidate_status { last_received { term: 4 index: 14235 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.786087 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.786201 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:12:02.798851 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 LEADER]: Stepping down as leader of term 4
I20240503 08:12:02.798967 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:02.799041 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14236, Committed index: 14236, Last appended: 4.14236, Last appended by leader: 14236, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.799163 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:02.804836 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:02.804929 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:02.805625 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.805799 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 5 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.806005 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 5 candidate_status { last_received { term: 4 index: 14236 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.805994 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 5 candidate_status { last_received { term: 4 index: 14236 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.806118 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 5. Current term is 7.
I20240503 08:12:02.806293 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 5 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 5. Current term is 7.
I20240503 08:12:02.806331 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240503 08:12:02.806397 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 5 FOLLOWER]: Advancing to term 7
I20240503 08:12:02.806901 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 5.
I20240503 08:12:02.807063 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 7 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 5. Current term is 7.
I20240503 08:12:02.837602 26390 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.837666 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.837745 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:12:02.837767 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:12:02.838522 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.838722 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 8 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.838948 26390 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.838968 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 8 candidate_status { last_received { term: 4 index: 14236 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.838985 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 8 candidate_status { last_received { term: 4 index: 14236 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.839067 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 6 FOLLOWER]: Advancing to term 8
I20240503 08:12:02.839092 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:12:02.839114 26390 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 6 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:02.839303 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 6 candidate_status { last_received { term: 4 index: 14236 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.839375 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 6. Current term is 8.
I20240503 08:12:02.839599 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 6 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 6. Current term is 8.
I20240503 08:12:02.839637 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240503 08:12:02.839711 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 8.
I20240503 08:12:02.839790 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 6 candidate_status { last_received { term: 4 index: 14236 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.839818 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 8.
I20240503 08:12:02.839874 26285 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 6. Current term is 8.
I20240503 08:12:02.839905 26390 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 8 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 6. Current term is 8.
I20240503 08:12:02.839910 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:02.840013 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 6 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 6. Current term is 8.
I20240503 08:12:02.840148 26407 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 FOLLOWER]: Leader election won for term 8
I20240503 08:12:02.840204 26407 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:02.840252 26407 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14236, Committed index: 14236, Last appended: 4.14236, Last appended by leader: 14236, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.840723 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 4 to 8, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 8 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:02.871198 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 8 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 4 index: 14235. Preceding OpId from leader: term: 8 index: 14237. (index mismatch)
I20240503 08:12:02.871500 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14237, Last known committed idx: 14235, Time since last communication: 0.000s
I20240503 08:12:02.873042 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 8 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 4 index: 14236. Preceding OpId from leader: term: 8 index: 14237. (index mismatch)
I20240503 08:12:02.873247 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14237, Last known committed idx: 14236, Time since last communication: 0.000s
I20240503 08:12:02.904933 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 8 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:02.905055 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 8 FOLLOWER]: Advancing to term 9
I20240503 08:12:02.905956 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.906152 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 9 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.906339 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 14238 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.906347 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 9 candidate_status { last_received { term: 8 index: 14238 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.906631 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.906749 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240503 08:12:02.908452 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 9 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 8. Current term is 9. Ops: []
I20240503 08:12:02.908699 26424 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 8.14238, Next index: 14239, Last known committed idx: 14238, Time since last communication: 0.000s
I20240503 08:12:02.942936 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.943125 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 9 FOLLOWER]: Advancing to term 10
I20240503 08:12:02.944373 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.944602 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 10 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.945020 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 14238 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.945065 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 10 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 8. Current term is 10. Ops: []
I20240503 08:12:02.945005 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 10 candidate_status { last_received { term: 8 index: 14238 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.945343 26398 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 8.14238, Next index: 14239, Last known committed idx: 14238, Time since last communication: 0.000s
I20240503 08:12:02.945562 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.945674 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240503 08:12:02.952507 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 10 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 8. Current term is 10. Ops: [8.14239-8.14239]
I20240503 08:12:02.952868 26424 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 8.14238, Next index: 14239, Last known committed idx: 14238, Time since last communication: 0.000s
I20240503 08:12:02.978159 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:02.978263 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 10 FOLLOWER]: Advancing to term 11
I20240503 08:12:02.979135 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.979315 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 11 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:02.979504 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 11 candidate_status { last_received { term: 8 index: 14238 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.979524 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 11 candidate_status { last_received { term: 8 index: 14238 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:02.979779 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:02.979913 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240503 08:12:02.980216 26407 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 LEADER]: Stepping down as leader of term 8
I20240503 08:12:02.980263 26407 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:02.980319 26407 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14239, Committed index: 14239, Last appended: 8.14239, Last appended by leader: 14239, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.980404 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 8 FOLLOWER]: Advancing to term 9
I20240503 08:12:02.983811 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 8 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:02.983872 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 8 FOLLOWER]: Advancing to term 9
I20240503 08:12:02.984434 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:02.984580 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 9 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:02.984766 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 9 candidate_status { last_received { term: 8 index: 14239 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:02.984850 26285 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 9. Current term is 11.
I20240503 08:12:02.984781 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 9 candidate_status { last_received { term: 8 index: 14239 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:02.984999 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 9 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 9. Current term is 11.
I20240503 08:12:02.985036 26173 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20240503 08:12:02.985110 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 9 FOLLOWER]: Advancing to term 11
I20240503 08:12:02.985733 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 11 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 9. Current term is 11.
I20240503 08:12:02.985767 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 9.
I20240503 08:12:03.020423 26398 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.020530 26398 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 9 FOLLOWER]: Advancing to term 10
I20240503 08:12:03.021157 26398 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.021332 26398 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 10 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.021521 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 10 candidate_status { last_received { term: 8 index: 14239 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.021520 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 10 candidate_status { last_received { term: 8 index: 14239 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.021632 26285 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 10. Current term is 11.
I20240503 08:12:03.021634 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 10. Current term is 11.
I20240503 08:12:03.021802 26098 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 10 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 10. Current term is 11.
I20240503 08:12:03.021843 26098 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20240503 08:12:03.021905 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 10 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 10. Current term is 11.
I20240503 08:12:03.022006 26398 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 10 FOLLOWER]: Advancing to term 11
I20240503 08:12:03.022742 26398 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 10. Current term is 11.
I20240503 08:12:03.030207 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.030283 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 11 FOLLOWER]: Advancing to term 12
I20240503 08:12:03.030900 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.031057 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 12 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.031241 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 12 candidate_status { last_received { term: 8 index: 14239 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.031253 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 12 candidate_status { last_received { term: 8 index: 14239 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.031356 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 11 FOLLOWER]: Advancing to term 12
I20240503 08:12:03.031399 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 11 FOLLOWER]: Advancing to term 12
I20240503 08:12:03.031996 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 12.
I20240503 08:12:03.032109 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 12.
I20240503 08:12:03.032300 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:03.032394 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 12 FOLLOWER]: Leader election won for term 12
I20240503 08:12:03.032459 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 12 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.032506 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14239, Committed index: 14239, Last appended: 8.14239, Last appended by leader: 14239, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.033035 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 8 to 12, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 12 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:03.061404 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 12 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 8 index: 14239. Preceding OpId from leader: term: 12 index: 14240. (index mismatch)
I20240503 08:12:03.061691 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14240, Last known committed idx: 14239, Time since last communication: 0.000s
I20240503 08:12:03.062848 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 12 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 8 index: 14238. Preceding OpId from leader: term: 12 index: 14241. (index mismatch)
I20240503 08:12:03.063052 26390 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14240, Last known committed idx: 14238, Time since last communication: 0.000s
I20240503 08:12:03.093729 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.093839 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 12 FOLLOWER]: Advancing to term 13
I20240503 08:12:03.094731 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.094873 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 13 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 12. Current term is 13. Ops: [12.14242-12.14242]
I20240503 08:12:03.094905 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 13 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.095017 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 13 candidate_status { last_received { term: 12 index: 14241 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.095070 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 13 candidate_status { last_received { term: 12 index: 14241 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.095232 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.095248 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 12.14241, Next index: 14242, Last known committed idx: 14241, Time since last communication: 0.000s
I20240503 08:12:03.095330 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240503 08:12:03.128326 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 13 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 12. Current term is 13. Ops: [12.14242-12.14242]
I20240503 08:12:03.128607 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 12.14241, Next index: 14242, Last known committed idx: 14241, Time since last communication: 0.000s
I20240503 08:12:03.135871 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.136011 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 13 FOLLOWER]: Advancing to term 14
I20240503 08:12:03.137050 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.137240 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 14 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.137470 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 14 candidate_status { last_received { term: 12 index: 14241 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.137471 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 14 candidate_status { last_received { term: 12 index: 14241 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.137897 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.137989 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240503 08:12:03.141757 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 12. Current term is 14. Ops: [12.14242-12.14242]
I20240503 08:12:03.142004 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 12.14241, Next index: 14242, Last known committed idx: 14241, Time since last communication: 0.000s
I20240503 08:12:03.142590 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 12. Current term is 14. Ops: [12.14242-12.14243]
I20240503 08:12:03.142814 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 12.14241, Next index: 14242, Last known committed idx: 14241, Time since last communication: 0.000s
I20240503 08:12:03.167982 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 12. Current term is 14. Ops: [12.14242-12.14243]
I20240503 08:12:03.168248 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 12.14241, Next index: 14242, Last known committed idx: 14241, Time since last communication: 0.000s
I20240503 08:12:03.173353 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.173420 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 12 FOLLOWER]: Advancing to term 13
I20240503 08:12:03.174085 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.174234 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 13 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.174435 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 13 candidate_status { last_received { term: 12 index: 14243 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.174438 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 13 candidate_status { last_received { term: 12 index: 14243 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.174547 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 13. Current term is 14.
I20240503 08:12:03.174721 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 13 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 13. Current term is 14.
I20240503 08:12:03.174757 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20240503 08:12:03.174813 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 13 FOLLOWER]: Advancing to term 14
I20240503 08:12:03.175419 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 13. Current term is 14.
I20240503 08:12:03.176760 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 14 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 12. Current term is 14. Ops: []
I20240503 08:12:03.176934 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 12.14243, Next index: 14244, Last known committed idx: 14243, Time since last communication: 0.000s
I20240503 08:12:03.179563 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 12 LEADER]: Stepping down as leader of term 12
I20240503 08:12:03.179608 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.179659 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14243, Committed index: 14243, Last appended: 12.14243, Last appended by leader: 14243, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.179734 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 12 FOLLOWER]: Advancing to term 13
I20240503 08:12:03.186772 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.186895 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 14 FOLLOWER]: Advancing to term 15
I20240503 08:12:03.187662 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.187822 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 15 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.188097 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 15 candidate_status { last_received { term: 12 index: 14241 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.188098 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 15 candidate_status { last_received { term: 12 index: 14241 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.188190 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 14 FOLLOWER]: Advancing to term 15
I20240503 08:12:03.188211 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 13 FOLLOWER]: Advancing to term 15
I20240503 08:12:03.188802 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 15 because replica has last-logged OpId of term: 12 index: 14243, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 14241.
I20240503 08:12:03.188987 26285 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 15 because replica has last-logged OpId of term: 12 index: 14243, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 14241.
I20240503 08:12:03.189170 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.189254 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20240503 08:12:03.213369 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.213460 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 15 FOLLOWER]: Advancing to term 16
I20240503 08:12:03.214139 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.214299 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 16 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.214475 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 16 candidate_status { last_received { term: 12 index: 14243 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.214547 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 15 FOLLOWER]: Advancing to term 16
I20240503 08:12:03.214480 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 16 candidate_status { last_received { term: 12 index: 14243 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.214634 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 15 FOLLOWER]: Advancing to term 16
I20240503 08:12:03.215101 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 16.
I20240503 08:12:03.215304 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:03.215392 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 16.
I20240503 08:12:03.215417 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 16 FOLLOWER]: Leader election won for term 16
I20240503 08:12:03.215468 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 16 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.215517 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14242, Committed index: 14242, Last appended: 12.14243, Last appended by leader: 14243, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.216099 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:03.221645 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 16 is higher.
W20240503 08:12:03.233842 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 16 is higher.
I20240503 08:12:03.240948 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 16 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 12 index: 14243. Preceding OpId from leader: term: 16 index: 14244. (index mismatch)
I20240503 08:12:03.241468 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14244, Last known committed idx: 14243, Time since last communication: 0.000s
I20240503 08:12:03.244248 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 16 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 12 index: 14241. Preceding OpId from leader: term: 16 index: 14244. (index mismatch)
I20240503 08:12:03.244683 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14244, Last known committed idx: 14241, Time since last communication: 0.000s
W20240503 08:12:03.262099 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 16 is higher.
W20240503 08:12:03.281518 26334 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 16 is higher.
I20240503 08:12:03.301751 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.301862 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 16 FOLLOWER]: Advancing to term 17
I20240503 08:12:03.302570 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.302716 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 17 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 16. Current term is 17. Ops: []
I20240503 08:12:03.302763 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 17 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.302923 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 17 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.302963 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 17 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.303165 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.303248 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20240503 08:12:03.303216 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 16.14245, Next index: 14246, Last known committed idx: 14245, Time since last communication: 0.000s
W20240503 08:12:03.314744 26334 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 16 is higher.
I20240503 08:12:03.327806 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 17 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 16. Current term is 17. Ops: []
I20240503 08:12:03.328097 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 16.14245, Next index: 14246, Last known committed idx: 14245, Time since last communication: 0.000s
I20240503 08:12:03.339071 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.339288 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 17 FOLLOWER]: Advancing to term 18
I20240503 08:12:03.340183 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.340371 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 18 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.340600 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 18 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.340623 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 18 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.340871 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.340963 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20240503 08:12:03.356381 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 18 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 16. Current term is 18. Ops: []
I20240503 08:12:03.356652 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 16.14245, Next index: 14246, Last known committed idx: 14245, Time since last communication: 0.000s
I20240503 08:12:03.373342 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 18 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 16. Current term is 18. Ops: [16.14246-16.14246]
I20240503 08:12:03.373596 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 16.14245, Next index: 14246, Last known committed idx: 14245, Time since last communication: 0.000s
I20240503 08:12:03.381510 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.381604 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 18 FOLLOWER]: Advancing to term 19
I20240503 08:12:03.382434 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.382603 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 19 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.382786 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 19 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.382831 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 19 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.383105 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.383193 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20240503 08:12:03.409601 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 16. Current term is 19. Ops: [16.14246-16.14246]
I20240503 08:12:03.409901 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 16.14245, Next index: 14246, Last known committed idx: 14245, Time since last communication: 0.000s
I20240503 08:12:03.414683 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 16 LEADER]: Stepping down as leader of term 16
I20240503 08:12:03.414748 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.414822 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14246, Committed index: 14246, Last appended: 16.14246, Last appended by leader: 14246, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.414927 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 16 FOLLOWER]: Advancing to term 17
I20240503 08:12:03.428339 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.428442 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 16 FOLLOWER]: Advancing to term 17
I20240503 08:12:03.429128 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.429296 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 17 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.429487 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 17 candidate_status { last_received { term: 16 index: 14246 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.429507 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 17 candidate_status { last_received { term: 16 index: 14246 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.429616 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 17. Current term is 19.
I20240503 08:12:03.429782 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 17 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 17. Current term is 19.
I20240503 08:12:03.429826 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20240503 08:12:03.429884 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 17 FOLLOWER]: Advancing to term 19
I20240503 08:12:03.430284 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 17.
I20240503 08:12:03.430538 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 19 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 17. Current term is 19.
I20240503 08:12:03.466918 26390 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.467028 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 17 FOLLOWER]: Advancing to term 18
I20240503 08:12:03.467991 26390 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.468178 26390 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 18 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.468385 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 18 candidate_status { last_received { term: 16 index: 14246 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.468396 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 18 candidate_status { last_received { term: 16 index: 14246 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.468488 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 18. Current term is 19.
I20240503 08:12:03.468499 26285 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 18. Current term is 19.
I20240503 08:12:03.468678 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 18 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 18. Current term is 19.
I20240503 08:12:03.468714 26173 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20240503 08:12:03.468748 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 18 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 18. Current term is 19.
I20240503 08:12:03.468884 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 18 FOLLOWER]: Advancing to term 19
I20240503 08:12:03.469403 26390 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 18. Current term is 19.
I20240503 08:12:03.472220 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.472292 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 19 FOLLOWER]: Advancing to term 20
I20240503 08:12:03.473182 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.473356 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 20 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.473536 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 20 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.473551 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 20 candidate_status { last_received { term: 16 index: 14245 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.473611 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 19 FOLLOWER]: Advancing to term 20
I20240503 08:12:03.473628 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 19 FOLLOWER]: Advancing to term 20
I20240503 08:12:03.474225 26285 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 20 because replica has last-logged OpId of term: 16 index: 14246, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14245.
I20240503 08:12:03.474268 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 20 because replica has last-logged OpId of term: 16 index: 14246, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14245.
I20240503 08:12:03.474287 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.474350 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 20 FOLLOWER]: Advancing to term 21
I20240503 08:12:03.474470 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.474586 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20240503 08:12:03.474996 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.475126 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 21 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.475258 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 21 candidate_status { last_received { term: 16 index: 14246 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.475313 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 20 FOLLOWER]: Advancing to term 21
I20240503 08:12:03.475912 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 21.
I20240503 08:12:03.476019 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 21 candidate_status { last_received { term: 16 index: 14246 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.476080 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 20 FOLLOWER]: Advancing to term 21
I20240503 08:12:03.476085 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:03.476207 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 21 FOLLOWER]: Leader election won for term 21
I20240503 08:12:03.476320 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 21 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:03.476372 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14246, Committed index: 14246, Last appended: 16.14246, Last appended by leader: 14246, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.476703 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 21.
I20240503 08:12:03.476955 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 16 to 21, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 21 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:03.500348 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 21 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 16 index: 14246. Preceding OpId from leader: term: 21 index: 14247. (index mismatch)
I20240503 08:12:03.500610 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14247, Last known committed idx: 14246, Time since last communication: 0.000s
I20240503 08:12:03.500872 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 21 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 16 index: 14245. Preceding OpId from leader: term: 21 index: 14247. (index mismatch)
I20240503 08:12:03.501049 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14247, Last known committed idx: 14245, Time since last communication: 0.000s
I20240503 08:12:03.533464 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:03.533586 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 21 FOLLOWER]: Advancing to term 22
I20240503 08:12:03.534283 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.534416 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 22 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 22. Ops: []
I20240503 08:12:03.534443 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 22 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.534600 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 22 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.534682 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 22 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.534749 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.534838 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.534909 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20240503 08:12:03.557416 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 22 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 22. Ops: []
I20240503 08:12:03.557700 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.577919 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.578027 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 22 FOLLOWER]: Advancing to term 23
I20240503 08:12:03.578739 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.578898 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 23 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.579134 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 23 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.579147 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 23 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.579420 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.579505 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20240503 08:12:03.583972 26390 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 21 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:03.584034 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 21 FOLLOWER]: Advancing to term 22
I20240503 08:12:03.584651 26390 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.584791 26390 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 22 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.584978 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.585052 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 22. Current term is 23.
I20240503 08:12:03.584956 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.585203 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 22 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 22. Current term is 23.
I20240503 08:12:03.585240 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20240503 08:12:03.585286 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 22 FOLLOWER]: Advancing to term 23
I20240503 08:12:03.585768 26390 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 22. Current term is 23.
I20240503 08:12:03.588008 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 23 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 23. Ops: []
I20240503 08:12:03.588213 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.588338 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 23. Ops: []
I20240503 08:12:03.588505 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.611972 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 23. Ops: []
I20240503 08:12:03.612291 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.620463 26390 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.620565 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 23 FOLLOWER]: Advancing to term 24
I20240503 08:12:03.620651 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 23. Ops: [21.14249-21.14249]
I20240503 08:12:03.620870 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.621486 26390 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.621639 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 21. Current term is 24. Ops: [21.14249-21.14249]
I20240503 08:12:03.621647 26390 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 24 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.621795 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.621876 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14248 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.621958 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 23 FOLLOWER]: Advancing to term 24
I20240503 08:12:03.622100 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 21.14248, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.622643 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 24.
I20240503 08:12:03.622799 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.622941 26390 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 FOLLOWER]: Leader election won for term 24
I20240503 08:12:03.622992 26390 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.623050 26390 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14248, Committed index: 14248, Last appended: 21.14248, Last appended by leader: 14248, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.623628 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 21 to 24, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 24 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:03.634261 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 24 is higher.
I20240503 08:12:03.648109 26399 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 21 LEADER]: Stepping down as leader of term 21
I20240503 08:12:03.648221 26399 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:03.648298 26399 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14248, Committed index: 14248, Last appended: 21.14249, Last appended by leader: 14249, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.648420 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 21 FOLLOWER]: Advancing to term 22
I20240503 08:12:03.656226 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 22 FOLLOWER]: Advancing to term 24
I20240503 08:12:03.657150 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 24 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14249. Preceding OpId from leader: term: 24 index: 14249. (term mismatch)
I20240503 08:12:03.657202 26285 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14248
I20240503 08:12:03.657222 26285 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted WRITE_OP operation due to leader change: 21.14249
I20240503 08:12:03.657677 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.658038 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 24 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14248. Preceding OpId from leader: term: 24 index: 14250. (index mismatch)
I20240503 08:12:03.658305 26390 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14249, Last known committed idx: 14248, Time since last communication: 0.000s
I20240503 08:12:03.687950 26348 mvcc.cc:205] Tried to move back new op lower bound from 7023509191301566464 to 7023509191160840192. Current Snapshot: MvccSnapshot[applied={T|T < 7023509191301566464 or (T in {7023509191301566464})}]
I20240503 08:12:03.719908 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.719998 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 24 FOLLOWER]: Advancing to term 25
I20240503 08:12:03.720660 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.720782 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 24. Current term is 25. Ops: []
I20240503 08:12:03.720821 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 25 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.720968 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 24.14251, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.721004 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 25 candidate_status { last_received { term: 24 index: 14251 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.720978 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 25 candidate_status { last_received { term: 24 index: 14251 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.721220 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.721315 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20240503 08:12:03.749411 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 24 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.749552 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 24 FOLLOWER]: Advancing to term 25
I20240503 08:12:03.750285 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.750413 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 25 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 24. Current term is 25. Ops: []
I20240503 08:12:03.750454 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 25 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.750655 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 25 candidate_status { last_received { term: 24 index: 14251 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.750730 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 25: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:03.750682 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 25 candidate_status { last_received { term: 24 index: 14251 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.750772 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 24.14251, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.750816 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 24. Current term is 25. Ops: []
I20240503 08:12:03.750922 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:03.750967 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 24.14251, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.751036 26473 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20240503 08:12:03.753880 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.753962 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 25 FOLLOWER]: Advancing to term 26
I20240503 08:12:03.754575 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.754758 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 26 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.754945 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 26 candidate_status { last_received { term: 24 index: 14251 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.754945 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 26 candidate_status { last_received { term: 24 index: 14251 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.755050 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 25 FOLLOWER]: Advancing to term 26
I20240503 08:12:03.755589 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 26.
I20240503 08:12:03.755761 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:03.755837 26407 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 FOLLOWER]: Leader election won for term 26
I20240503 08:12:03.755942 26407 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:03.755988 26407 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14251, Committed index: 14251, Last appended: 24.14251, Last appended by leader: 14251, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.756376 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 24 to 26, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 26 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:03.772212 26210 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 LEADER]: Stepping down as leader of term 24
I20240503 08:12:03.772271 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 24. Current term is 26. Ops: [24.14252-24.14252]
I20240503 08:12:03.772229 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 26 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 24 index: 14251. Preceding OpId from leader: term: 26 index: 14253. (index mismatch)
I20240503 08:12:03.772425 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 26 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 24. Current term is 26. Ops: [24.14252-24.14252]
I20240503 08:12:03.772455 26390 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 24.14251, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.772536 26390 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 26.14253 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14251
I20240503 08:12:03.772575 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 24.14251, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.772307 26210 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.772547 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.772709 26210 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14251, Committed index: 14251, Last appended: 24.14252, Last appended by leader: 14252, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.772795 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 24 FOLLOWER]: Advancing to term 26
I20240503 08:12:03.773650 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 26 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 24 index: 14252. Preceding OpId from leader: term: 26 index: 14253. (index mismatch)
W20240503 08:12:03.773782 26384 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 26 is higher.
I20240503 08:12:03.773977 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14252, Last known committed idx: 14251, Time since last communication: 0.000s
I20240503 08:12:03.774056 26407 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 24.14252 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14251
I20240503 08:12:03.774262 26210 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting all ops after (but not including) 14251
I20240503 08:12:03.774302 26210 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 24.14252
I20240503 08:12:03.805212 26390 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 26 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:03.805303 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 26 FOLLOWER]: Advancing to term 27
I20240503 08:12:03.805982 26390 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.806161 26390 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 27 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.806373 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14253 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.806373 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14253 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.806670 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.806761 26390 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20240503 08:12:03.807600 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 26. Current term is 27. Ops: []
I20240503 08:12:03.807834 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 26.14253, Next index: 14254, Last known committed idx: 14253, Time since last communication: 0.000s
I20240503 08:12:03.826349 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 26 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:03.826458 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 26 FOLLOWER]: Advancing to term 27
I20240503 08:12:03.827173 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.827333 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 27 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.827541 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 27 candidate_status { last_received { term: 26 index: 14253 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.827546 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 27 candidate_status { last_received { term: 26 index: 14253 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.827697 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 27: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:03.827833 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:03.827915 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20240503 08:12:03.830821 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 27 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 26. Current term is 27. Ops: []
I20240503 08:12:03.831053 26407 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 26.14253, Next index: 14254, Last known committed idx: 14253, Time since last communication: 0.000s
I20240503 08:12:03.837594 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 26. Current term is 27. Ops: []
I20240503 08:12:03.837874 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 26.14253, Next index: 14254, Last known committed idx: 14253, Time since last communication: 0.000s
I20240503 08:12:03.843122 26494 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 LEADER]: Stepping down as leader of term 26
I20240503 08:12:03.843211 26494 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:03.843286 26494 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14253, Committed index: 14253, Last appended: 26.14253, Last appended by leader: 14253, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.843392 26494 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 26 FOLLOWER]: Advancing to term 27
I20240503 08:12:03.844281 26390 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.844347 26390 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 27 FOLLOWER]: Advancing to term 28
I20240503 08:12:03.844964 26390 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.845116 26390 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 28 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:03.845327 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14253 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.845330 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14253 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.845398 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 27 FOLLOWER]: Advancing to term 28
I20240503 08:12:03.845445 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 27 FOLLOWER]: Advancing to term 28
I20240503 08:12:03.846004 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 28.
I20240503 08:12:03.846011 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 28.
I20240503 08:12:03.846191 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:03.846276 26390 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 28 FOLLOWER]: Leader election won for term 28
I20240503 08:12:03.846343 26390 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 28 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.846377 26390 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14253, Committed index: 14253, Last appended: 26.14253, Last appended by leader: 14253, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.846907 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 26 to 28, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 28 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:03.878482 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 28 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 26 index: 14253. Preceding OpId from leader: term: 28 index: 14254. (index mismatch)
I20240503 08:12:03.878759 26390 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14254, Last known committed idx: 14253, Time since last communication: 0.000s
I20240503 08:12:03.879308 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 28 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 26 index: 14253. Preceding OpId from leader: term: 28 index: 14254. (index mismatch)
I20240503 08:12:03.879488 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14254, Last known committed idx: 14253, Time since last communication: 0.000s
W20240503 08:12:03.896564 26384 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 28 is higher.
I20240503 08:12:03.910168 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 28 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.910266 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 28 FOLLOWER]: Advancing to term 29
I20240503 08:12:03.910964 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.911114 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 29 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.911334 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 29 candidate_status { last_received { term: 28 index: 14254 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.911334 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 29 candidate_status { last_received { term: 28 index: 14254 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.911621 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:03.911710 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20240503 08:12:03.914175 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 29 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 28. Current term is 29. Ops: []
I20240503 08:12:03.914446 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 28.14254, Next index: 14255, Last known committed idx: 14254, Time since last communication: 0.000s
I20240503 08:12:03.932732 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:03.932845 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 28 FOLLOWER]: Advancing to term 29
I20240503 08:12:03.933642 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.933804 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 29 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.934049 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 29 candidate_status { last_received { term: 28 index: 14254 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:03.934058 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 29 candidate_status { last_received { term: 28 index: 14254 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.934126 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 29: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:03.934299 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:03.934404 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20240503 08:12:03.936784 26384 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 28 is higher.
I20240503 08:12:03.937201 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 29 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 28. Current term is 29. Ops: []
I20240503 08:12:03.937376 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 28.14254, Next index: 14255, Last known committed idx: 14254, Time since last communication: 0.000s
I20240503 08:12:03.947607 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:03.947710 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 29 FOLLOWER]: Advancing to term 30
I20240503 08:12:03.948410 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.948582 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 30 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:03.948827 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 30 candidate_status { last_received { term: 28 index: 14254 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:03.948850 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 30 candidate_status { last_received { term: 28 index: 14254 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:03.948962 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 29 FOLLOWER]: Advancing to term 30
I20240503 08:12:03.949025 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 28. Current term is 30. Ops: []
I20240503 08:12:03.949287 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 28.14254, Next index: 14255, Last known committed idx: 14254, Time since last communication: 0.000s
I20240503 08:12:03.949640 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 30.
I20240503 08:12:03.949812 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:03.949947 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 FOLLOWER]: Leader election won for term 30
I20240503 08:12:03.950012 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:03.950057 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14254, Committed index: 14254, Last appended: 28.14254, Last appended by leader: 14254, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.950541 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 28 to 30, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 30 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:03.969306 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 30 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 28. Current term is 30. Ops: []
I20240503 08:12:03.969635 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 28.14254, Next index: 14255, Last known committed idx: 14254, Time since last communication: 0.000s
I20240503 08:12:03.975016 26210 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 28 LEADER]: Stepping down as leader of term 28
I20240503 08:12:03.975096 26210 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:03.975143 26210 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14254, Committed index: 14254, Last appended: 28.14254, Last appended by leader: 14254, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:03.975265 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 28 FOLLOWER]: Advancing to term 30
I20240503 08:12:03.975939 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 30 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 28 index: 14254. Preceding OpId from leader: term: 30 index: 14255. (index mismatch)
I20240503 08:12:03.976001 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 30 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 28 index: 14254. Preceding OpId from leader: term: 30 index: 14255. (index mismatch)
I20240503 08:12:03.976219 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14255, Last known committed idx: 14254, Time since last communication: 0.000s
I20240503 08:12:03.976307 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14255, Last known committed idx: 14254, Time since last communication: 0.000s
I20240503 08:12:04.007723 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 30 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.007822 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 30 FOLLOWER]: Advancing to term 31
I20240503 08:12:04.008525 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.008697 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 31 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.008900 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 31 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.008900 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 31 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.009133 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.009258 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20240503 08:12:04.013831 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 31 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 30. Current term is 31. Ops: []
I20240503 08:12:04.014143 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 30.14255, Next index: 14256, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.044919 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.045027 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 31 FOLLOWER]: Advancing to term 32
I20240503 08:12:04.045748 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.045926 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 32 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.046170 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 32 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.046171 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 32 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.046427 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.046536 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20240503 08:12:04.049036 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 30. Current term is 32. Ops: []
I20240503 08:12:04.049391 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 30.14255, Next index: 14256, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.056636 26399 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 LEADER]: Stepping down as leader of term 30
I20240503 08:12:04.056718 26399 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.056785 26399 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14255, Committed index: 14255, Last appended: 30.14255, Last appended by leader: 14255, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.056905 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 30 FOLLOWER]: Advancing to term 31
I20240503 08:12:04.087071 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 30 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.087186 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 30 FOLLOWER]: Advancing to term 31
I20240503 08:12:04.087860 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.088032 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 31 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.088263 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 31 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.088258 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 31 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.088351 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 31. Current term is 32.
I20240503 08:12:04.088533 26098 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 31 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 31. Current term is 32.
I20240503 08:12:04.088570 26098 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20240503 08:12:04.088655 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 31 FOLLOWER]: Advancing to term 32
I20240503 08:12:04.088959 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 31.
I20240503 08:12:04.089215 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 31. Current term is 32.
I20240503 08:12:04.089293 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.089344 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 32 FOLLOWER]: Advancing to term 33
I20240503 08:12:04.089802 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.089915 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 33 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.090094 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 33 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.090137 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 33 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.090143 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 31 FOLLOWER]: Advancing to term 33
I20240503 08:12:04.090224 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 32 FOLLOWER]: Advancing to term 33
I20240503 08:12:04.090687 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 33.
I20240503 08:12:04.090847 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 33.
I20240503 08:12:04.090886 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:04.090958 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 33 FOLLOWER]: Leader election won for term 33
I20240503 08:12:04.091025 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 33 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:04.091081 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14255, Committed index: 14255, Last appended: 30.14255, Last appended by leader: 14255, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.091703 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 30 to 33, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 33 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:04.093047 26384 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 30 current term: 33 is higher.
I20240503 08:12:04.126111 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 33 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 30 index: 14255. Preceding OpId from leader: term: 33 index: 14256. (index mismatch)
I20240503 08:12:04.126485 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14256, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.127550 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 33 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 30 index: 14255. Preceding OpId from leader: term: 33 index: 14256. (index mismatch)
I20240503 08:12:04.127753 26481 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.127820 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 33 FOLLOWER]: Advancing to term 34
I20240503 08:12:04.127755 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14256, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.128576 26481 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.128681 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 33. Current term is 34. Ops: [33.14256-33.14256]
I20240503 08:12:04.128713 26481 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 34 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.128813 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 34 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.128859 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 30.14255, Next index: 14256, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.128916 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 34 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.129101 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.129187 26481 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20240503 08:12:04.157331 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 33. Current term is 34. Ops: [33.14256-33.14256]
I20240503 08:12:04.157693 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 30.14255, Next index: 14256, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.158385 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:04.158442 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 33 FOLLOWER]: Advancing to term 34
I20240503 08:12:04.159089 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.159238 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 34 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.159408 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 34 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.159488 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 34: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:04.159471 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 34 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.159651 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.159786 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20240503 08:12:04.160777 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.160837 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 34 FOLLOWER]: Advancing to term 35
I20240503 08:12:04.161365 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 34 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 33. Current term is 34. Ops: []
I20240503 08:12:04.161530 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.161613 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 33.14256, Next index: 14257, Last known committed idx: 14256, Time since last communication: 0.000s
I20240503 08:12:04.161685 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 35 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.161811 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 35 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.161851 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 35 candidate_status { last_received { term: 30 index: 14255 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.161885 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 34 FOLLOWER]: Advancing to term 35
I20240503 08:12:04.162451 26134 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for term 35 because replica has last-logged OpId of term: 33 index: 14256, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 14255.
I20240503 08:12:04.162658 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.162753 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20240503 08:12:04.168776 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 33 LEADER]: Stepping down as leader of term 33
I20240503 08:12:04.168849 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:04.168911 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14256, Committed index: 14256, Last appended: 33.14256, Last appended by leader: 14256, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.169021 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 33 FOLLOWER]: Advancing to term 34
I20240503 08:12:04.198530 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.198668 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 35 FOLLOWER]: Advancing to term 36
I20240503 08:12:04.199358 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.199522 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 36 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.199744 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 36 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.199755 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 36 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.199827 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 34 FOLLOWER]: Advancing to term 36
I20240503 08:12:04.199844 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 35 FOLLOWER]: Advancing to term 36
I20240503 08:12:04.200371 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 36.
I20240503 08:12:04.200350 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 36.
I20240503 08:12:04.200577 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:04.200678 26407 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 36 FOLLOWER]: Leader election won for term 36
I20240503 08:12:04.200752 26407 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 36 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:04.200785 26407 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14256, Committed index: 14256, Last appended: 33.14256, Last appended by leader: 14256, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.201359 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 33 to 36, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 36 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.223928 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 36 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 30 index: 14255. Preceding OpId from leader: term: 36 index: 14257. (index mismatch)
I20240503 08:12:04.224262 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14257, Last known committed idx: 14255, Time since last communication: 0.000s
I20240503 08:12:04.233171 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.233275 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 36 FOLLOWER]: Advancing to term 37
I20240503 08:12:04.234048 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.234230 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 37 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.234462 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 37 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.234476 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 37 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.234716 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.234812 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20240503 08:12:04.234933 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 37 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 36. Current term is 37. Ops: []
I20240503 08:12:04.235410 26407 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14257, Last known committed idx: 14256, Time since last communication: 0.000s
I20240503 08:12:04.261543 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 37 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 36. Current term is 37. Ops: [36.14257-36.14257]
I20240503 08:12:04.261832 26494 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 33.14256, Next index: 14257, Last known committed idx: 14256, Time since last communication: 0.000s
I20240503 08:12:04.267225 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.267290 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 37 FOLLOWER]: Advancing to term 38
I20240503 08:12:04.267959 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.268113 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 38 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.268325 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 38 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.268332 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 38 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.268656 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.268769 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20240503 08:12:04.292820 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 36. Current term is 38. Ops: [36.14257-36.14257]
I20240503 08:12:04.293198 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 33.14256, Next index: 14257, Last known committed idx: 14256, Time since last communication: 0.000s
I20240503 08:12:04.306696 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 36 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:04.306819 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 36 FOLLOWER]: Advancing to term 37
I20240503 08:12:04.307583 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.307782 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 37 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.307955 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 37 candidate_status { last_received { term: 36 index: 14257 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.307946 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 37 candidate_status { last_received { term: 36 index: 14257 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.308080 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 37. Current term is 38.
I20240503 08:12:04.308233 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 37 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 37. Current term is 38.
I20240503 08:12:04.308270 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
I20240503 08:12:04.308347 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 37 FOLLOWER]: Advancing to term 38
I20240503 08:12:04.308933 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 37. Current term is 38.
I20240503 08:12:04.310096 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 38 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 36. Current term is 38. Ops: []
I20240503 08:12:04.310307 26494 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 36.14257, Next index: 14258, Last known committed idx: 14257, Time since last communication: 0.000s
I20240503 08:12:04.318583 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.318715 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 38 FOLLOWER]: Advancing to term 39
I20240503 08:12:04.319332 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.319505 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 39 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.319751 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 39 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.319751 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 39 candidate_status { last_received { term: 33 index: 14256 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.319869 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 38 FOLLOWER]: Advancing to term 39
I20240503 08:12:04.320488 26284 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 39 because replica has last-logged OpId of term: 36 index: 14257, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 14256.
I20240503 08:12:04.320703 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.320797 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20240503 08:12:04.322957 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 39 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 36. Current term is 39. Ops: [36.14257-36.14257]
I20240503 08:12:04.323299 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 33.14256, Next index: 14257, Last known committed idx: 14256, Time since last communication: 0.000s
I20240503 08:12:04.343510 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.343561 26407 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 36 LEADER]: Stepping down as leader of term 36
I20240503 08:12:04.343626 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 39 FOLLOWER]: Advancing to term 40
I20240503 08:12:04.343648 26407 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:04.343727 26407 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14257, Committed index: 14257, Last appended: 36.14257, Last appended by leader: 14257, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.343837 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 36 FOLLOWER]: Advancing to term 37
I20240503 08:12:04.344478 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.344632 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 36. Current term is 40. Ops: []
I20240503 08:12:04.344650 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 40 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.344812 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 40 candidate_status { last_received { term: 36 index: 14257 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.344898 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 39 FOLLOWER]: Advancing to term 40
I20240503 08:12:04.344916 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 40 candidate_status { last_received { term: 36 index: 14257 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.345000 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 37 FOLLOWER]: Advancing to term 40
I20240503 08:12:04.345482 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 40.
I20240503 08:12:04.345777 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:04.345788 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 40.
I20240503 08:12:04.345923 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 FOLLOWER]: Leader election won for term 40
I20240503 08:12:04.345994 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.346045 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14257, Committed index: 14257, Last appended: 36.14257, Last appended by leader: 14257, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.346658 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 36 to 40, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 40 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.368791 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 40 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 33 index: 14256. Preceding OpId from leader: term: 40 index: 14258. (index mismatch)
I20240503 08:12:04.369092 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14258, Last known committed idx: 14256, Time since last communication: 0.000s
I20240503 08:12:04.369900 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 40 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 36 index: 14257. Preceding OpId from leader: term: 40 index: 14258. (index mismatch)
I20240503 08:12:04.370149 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14258, Last known committed idx: 14257, Time since last communication: 0.000s
I20240503 08:12:04.423692 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 40 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.423791 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 40 FOLLOWER]: Advancing to term 41
I20240503 08:12:04.424412 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.424542 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 40. Current term is 41. Ops: []
I20240503 08:12:04.424633 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 41 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.424785 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 41 candidate_status { last_received { term: 40 index: 14258 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.424809 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 40.14258, Next index: 14259, Last known committed idx: 14258, Time since last communication: 0.000s
I20240503 08:12:04.424809 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 41 candidate_status { last_received { term: 40 index: 14258 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.425091 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.425185 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20240503 08:12:04.451629 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 40. Current term is 41. Ops: []
I20240503 08:12:04.451942 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 40.14258, Next index: 14259, Last known committed idx: 14258, Time since last communication: 0.000s
I20240503 08:12:04.453224 26494 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 40 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.453295 26494 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 40 FOLLOWER]: Advancing to term 41
I20240503 08:12:04.454212 26494 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.454375 26494 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 41 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.454556 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 41 candidate_status { last_received { term: 40 index: 14258 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.454766 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 41 candidate_status { last_received { term: 40 index: 14258 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.454842 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 41 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 40. Current term is 41. Ops: []
I20240503 08:12:04.454854 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 41: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:04.455049 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 40.14258, Next index: 14259, Last known committed idx: 14258, Time since last communication: 0.000s
I20240503 08:12:04.455078 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.455207 26494 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20240503 08:12:04.462733 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.462806 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 41 FOLLOWER]: Advancing to term 42
I20240503 08:12:04.463485 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.463637 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 42 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.463871 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 42 candidate_status { last_received { term: 40 index: 14258 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.463873 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 42 candidate_status { last_received { term: 40 index: 14258 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.463956 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 41 FOLLOWER]: Advancing to term 42
I20240503 08:12:04.464497 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 42.
I20240503 08:12:04.464716 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.464841 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 42 FOLLOWER]: Leader election won for term 42
I20240503 08:12:04.464905 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 42 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:04.464959 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14258, Committed index: 14258, Last appended: 40.14258, Last appended by leader: 14258, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.465415 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 40 to 42, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 42 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:04.466060 26399 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 LEADER]: Stepping down as leader of term 40
I20240503 08:12:04.466112 26399 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.466182 26399 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14258, Committed index: 14258, Last appended: 40.14258, Last appended by leader: 14258, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.466266 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 40 FOLLOWER]: Advancing to term 41
I20240503 08:12:04.492921 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 41 FOLLOWER]: Advancing to term 42
I20240503 08:12:04.493693 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 42 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 40 index: 14258. Preceding OpId from leader: term: 42 index: 14259. (index mismatch)
I20240503 08:12:04.493948 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14259, Last known committed idx: 14258, Time since last communication: 0.000s
I20240503 08:12:04.501979 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 42 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 40 index: 14258. Preceding OpId from leader: term: 42 index: 14259. (index mismatch)
I20240503 08:12:04.502225 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14259, Last known committed idx: 14258, Time since last communication: 0.000s
I20240503 08:12:04.551476 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 42 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:04.551582 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 42 FOLLOWER]: Advancing to term 43
I20240503 08:12:04.552312 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.552563 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 43 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.552642 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 43 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 42. Current term is 43. Ops: []
I20240503 08:12:04.552700 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 43 candidate_status { last_received { term: 42 index: 14259 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.552826 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 42.14259, Next index: 14260, Last known committed idx: 14259, Time since last communication: 0.000s
I20240503 08:12:04.552942 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 43 candidate_status { last_received { term: 42 index: 14259 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.553140 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.553239 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20240503 08:12:04.567070 26384 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 42 LEADER]: Stepping down as leader of term 42
I20240503 08:12:04.567242 26384 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:04.567320 26384 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14259, Committed index: 14259, Last appended: 42.14259, Last appended by leader: 14259, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.567463 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 42 FOLLOWER]: Advancing to term 43
I20240503 08:12:04.583165 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:04.583276 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 42 FOLLOWER]: Advancing to term 43
I20240503 08:12:04.584026 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.584198 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 43 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.584422 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 43 candidate_status { last_received { term: 42 index: 14259 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.584422 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 43 candidate_status { last_received { term: 42 index: 14259 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.584667 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 43: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:04.585323 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 43.
I20240503 08:12:04.585566 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.585687 26407 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 43 FOLLOWER]: Leader election won for term 43
I20240503 08:12:04.585744 26407 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 43 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:04.585795 26407 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14259, Committed index: 14259, Last appended: 42.14259, Last appended by leader: 14259, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.586306 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 42 to 43, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 43 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.589037 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.589093 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 43 FOLLOWER]: Advancing to term 44
I20240503 08:12:04.589634 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.589762 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 44 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.589933 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 44 candidate_status { last_received { term: 42 index: 14259 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.589933 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 44 candidate_status { last_received { term: 42 index: 14259 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.590047 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 43 FOLLOWER]: Advancing to term 44
I20240503 08:12:04.590689 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 44.
I20240503 08:12:04.590867 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.590997 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 FOLLOWER]: Leader election won for term 44
I20240503 08:12:04.591079 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.591137 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14259, Committed index: 14259, Last appended: 42.14259, Last appended by leader: 14259, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.610236 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 43 to 44, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 44 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.615073 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 44 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 42 index: 14259. Preceding OpId from leader: term: 44 index: 14260. (index mismatch)
I20240503 08:12:04.615355 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14260, Last known committed idx: 14259, Time since last communication: 0.000s
I20240503 08:12:04.618603 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 43. Current term is 44. Ops: []
I20240503 08:12:04.618870 26407 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14260, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.618965 26407 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 44.14260 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14260
I20240503 08:12:04.620607 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 44 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 43. Current term is 44. Ops: []
I20240503 08:12:04.620803 26407 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14260, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.626242 26134 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 43 LEADER]: Stepping down as leader of term 43
I20240503 08:12:04.626302 26134 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:04.626379 26134 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14259, Committed index: 14259, Last appended: 43.14260, Last appended by leader: 14260, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.626480 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 43 FOLLOWER]: Advancing to term 44
I20240503 08:12:04.627140 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 44 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 43 index: 14260. Preceding OpId from leader: term: 44 index: 14260. (term mismatch)
I20240503 08:12:04.627182 26134 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14259
I20240503 08:12:04.627207 26134 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 43.14260
I20240503 08:12:04.627231 26134 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:04.627481 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14260, Last known committed idx: 14259, Time since last communication: 0.000s
I20240503 08:12:04.658131 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 44 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.658236 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 44 FOLLOWER]: Advancing to term 45
I20240503 08:12:04.659040 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.659191 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 45 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.659408 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 45 candidate_status { last_received { term: 44 index: 14260 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.659408 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 45 candidate_status { last_received { term: 44 index: 14260 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.659677 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.659765 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20240503 08:12:04.663378 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 44. Current term is 45. Ops: []
I20240503 08:12:04.663616 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 44.14260, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.672770 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 44 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.672857 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 44 FOLLOWER]: Advancing to term 45
I20240503 08:12:04.673522 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.673673 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 45 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.673888 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 45 candidate_status { last_received { term: 44 index: 14260 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.673888 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 45 candidate_status { last_received { term: 44 index: 14260 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.674058 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 45: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:04.674247 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.674331 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20240503 08:12:04.676708 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 45 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 44. Current term is 45. Ops: []
I20240503 08:12:04.676927 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 44.14260, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.696512 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 44. Current term is 45. Ops: []
I20240503 08:12:04.696774 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 44.14260, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.700274 26494 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.700340 26494 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 45 FOLLOWER]: Advancing to term 46
I20240503 08:12:04.701011 26494 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.701143 26494 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 46 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.701345 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 46 candidate_status { last_received { term: 44 index: 14260 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.701418 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 45 FOLLOWER]: Advancing to term 46
I20240503 08:12:04.701345 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 46 candidate_status { last_received { term: 44 index: 14260 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.701992 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 46.
I20240503 08:12:04.702189 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.702278 26494 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 FOLLOWER]: Leader election won for term 46
I20240503 08:12:04.702328 26494 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:04.702389 26494 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14260, Committed index: 14260, Last appended: 44.14260, Last appended by leader: 14260, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.702826 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 44 to 46, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 46 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:04.706794 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 46 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 44. Current term is 46. Ops: []
I20240503 08:12:04.707051 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 44.14260, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.721676 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 44. Current term is 46. Ops: []
I20240503 08:12:04.721971 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 44.14260, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.722090 26399 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 46.14261 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14260
I20240503 08:12:04.722506 26481 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 LEADER]: Stepping down as leader of term 44
I20240503 08:12:04.722550 26481 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.722635 26481 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14260, Committed index: 14260, Last appended: 44.14260, Last appended by leader: 14260, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.722718 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 44 FOLLOWER]: Advancing to term 45
I20240503 08:12:04.730657 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 46 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 44 index: 14260. Preceding OpId from leader: term: 46 index: 14261. (index mismatch)
I20240503 08:12:04.730981 26494 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.735821 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 45 FOLLOWER]: Advancing to term 46
I20240503 08:12:04.736584 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 46 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 44 index: 14260. Preceding OpId from leader: term: 46 index: 14261. (index mismatch)
I20240503 08:12:04.736842 26494 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14261, Last known committed idx: 14260, Time since last communication: 0.000s
I20240503 08:12:04.791147 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 46 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:04.791250 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 46 FOLLOWER]: Advancing to term 47
I20240503 08:12:04.791934 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.792100 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 47 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.792296 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 47 candidate_status { last_received { term: 46 index: 14261 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.792308 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 47 candidate_status { last_received { term: 46 index: 14261 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.792543 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.792635 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20240503 08:12:04.796379 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 47 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 46. Current term is 47. Ops: []
I20240503 08:12:04.796605 26494 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 46.14261, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
I20240503 08:12:04.796931 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 46 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:04.796972 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 46 FOLLOWER]: Advancing to term 47
I20240503 08:12:04.797573 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.797701 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 47 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.797840 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 47 candidate_status { last_received { term: 46 index: 14261 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.797848 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 47 candidate_status { last_received { term: 46 index: 14261 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.797935 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 47: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:04.798146 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.798377 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20240503 08:12:04.798467 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 47 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 46. Current term is 47. Ops: []
I20240503 08:12:04.798684 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 46.14261, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
W20240503 08:12:04.800256 26349 lock_manager.cc:494] Txn 4 has not acquired the partition lock after 1000ms
I20240503 08:12:04.827528 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 47 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 46. Current term is 47. Ops: []
I20240503 08:12:04.827518 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 47 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 46. Current term is 47. Ops: []
I20240503 08:12:04.827843 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 46.14261, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
I20240503 08:12:04.828088 26494 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 46.14261, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
I20240503 08:12:04.833052 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.833146 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 47 FOLLOWER]: Advancing to term 48
I20240503 08:12:04.833822 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.833984 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 48 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.834194 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 48 candidate_status { last_received { term: 46 index: 14261 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.834203 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 48 candidate_status { last_received { term: 46 index: 14261 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.834260 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 47 FOLLOWER]: Advancing to term 48
I20240503 08:12:04.834978 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 48.
I20240503 08:12:04.835177 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.835265 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 48 FOLLOWER]: Leader election won for term 48
I20240503 08:12:04.835319 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 48 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.835377 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14261, Committed index: 14261, Last appended: 46.14261, Last appended by leader: 14261, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.835847 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 46 to 48, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 48 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:04.841955 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 48 is higher.
I20240503 08:12:04.849790 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 48 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 46. Current term is 48. Ops: []
I20240503 08:12:04.850068 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 46.14261, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
I20240503 08:12:04.852294 26407 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 LEADER]: Stepping down as leader of term 46
I20240503 08:12:04.852376 26407 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:04.852443 26407 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14261, Committed index: 14261, Last appended: 46.14261, Last appended by leader: 14261, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.852566 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 46 FOLLOWER]: Advancing to term 47
I20240503 08:12:04.865325 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 48 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 46 index: 14261. Preceding OpId from leader: term: 48 index: 14262. (index mismatch)
I20240503 08:12:04.865736 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
I20240503 08:12:04.866431 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 47 FOLLOWER]: Advancing to term 48
I20240503 08:12:04.867525 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 48 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 46 index: 14261. Preceding OpId from leader: term: 48 index: 14262. (index mismatch)
I20240503 08:12:04.867738 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14262, Last known committed idx: 14261, Time since last communication: 0.000s
W20240503 08:12:04.872138 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 48 is higher.
I20240503 08:12:04.898022 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 48 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.898118 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 48 FOLLOWER]: Advancing to term 49
I20240503 08:12:04.898247 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 48 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:04.898313 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 48 FOLLOWER]: Advancing to term 49
I20240503 08:12:04.898866 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.899031 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 49 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.898985 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.899191 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 49 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.899268 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 49 candidate_status { last_received { term: 48 index: 14262 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.899317 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 49 candidate_status { last_received { term: 48 index: 14262 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.899351 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 49: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:04.899255 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 48. Current term is 49. Ops: []
I20240503 08:12:04.899248 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 49 candidate_status { last_received { term: 48 index: 14262 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.899406 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 49 candidate_status { last_received { term: 48 index: 14262 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.899546 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 49: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:04.899580 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.899688 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.899714 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20240503 08:12:04.899717 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 48.14262, Next index: 14263, Last known committed idx: 14262, Time since last communication: 0.000s
I20240503 08:12:04.899775 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20240503 08:12:04.905187 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 49 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 48. Current term is 49. Ops: []
I20240503 08:12:04.905406 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 48.14262, Next index: 14263, Last known committed idx: 14262, Time since last communication: 0.000s
I20240503 08:12:04.928288 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 48. Current term is 49. Ops: []
I20240503 08:12:04.928635 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 48.14262, Next index: 14263, Last known committed idx: 14262, Time since last communication: 0.000s
I20240503 08:12:04.932246 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:04.932317 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 49 FOLLOWER]: Advancing to term 50
I20240503 08:12:04.932986 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.933141 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 50 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:04.933357 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 50 candidate_status { last_received { term: 48 index: 14262 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.933429 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 49 FOLLOWER]: Advancing to term 50
I20240503 08:12:04.933362 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 50 candidate_status { last_received { term: 48 index: 14262 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:04.934046 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 50.
I20240503 08:12:04.934231 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:04.934314 26384 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 50 FOLLOWER]: Leader election won for term 50
I20240503 08:12:04.934367 26384 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 50 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:04.934428 26384 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14262, Committed index: 14262, Last appended: 48.14262, Last appended by leader: 14262, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.935127 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 48 to 50, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 50 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:04.937860 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 50 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 48. Current term is 50. Ops: []
I20240503 08:12:04.938112 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 48.14262, Next index: 14263, Last known committed idx: 14262, Time since last communication: 0.000s
I20240503 08:12:04.944900 26473 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 48 LEADER]: Stepping down as leader of term 48
I20240503 08:12:04.945003 26473 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 48 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:04.945079 26473 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14262, Committed index: 14262, Last appended: 48.14262, Last appended by leader: 14262, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.945201 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 48 FOLLOWER]: Advancing to term 49
I20240503 08:12:04.958699 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 49 FOLLOWER]: Advancing to term 50
I20240503 08:12:04.959439 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 50 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 48 index: 14262. Preceding OpId from leader: term: 50 index: 14263. (index mismatch)
I20240503 08:12:04.959707 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14263, Last known committed idx: 14262, Time since last communication: 0.000s
I20240503 08:12:04.962380 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 50 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 48 index: 14262. Preceding OpId from leader: term: 50 index: 14263. (index mismatch)
I20240503 08:12:04.962633 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14263, Last known committed idx: 14262, Time since last communication: 0.000s
I20240503 08:12:04.990872 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 50 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:04.990971 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 50 FOLLOWER]: Advancing to term 51
I20240503 08:12:04.991672 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:04.991815 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 51 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:04.992007 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 51 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:04.992007 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 51 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:04.992269 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:04.992362 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20240503 08:12:04.994814 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 51 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 50. Current term is 51. Ops: []
I20240503 08:12:04.995034 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 50.14263, Next index: 14264, Last known committed idx: 14263, Time since last communication: 0.000s
I20240503 08:12:05.017689 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 51 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 50. Current term is 51. Ops: []
I20240503 08:12:05.017999 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 50.14263, Next index: 14264, Last known committed idx: 14263, Time since last communication: 0.000s
I20240503 08:12:05.025199 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.025285 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 51 FOLLOWER]: Advancing to term 52
I20240503 08:12:05.025907 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.026048 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 52 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.026249 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 52 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.026273 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 52 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.026521 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:05.026635 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20240503 08:12:05.046964 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:05.047022 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 50 LEADER]: Stepping down as leader of term 50
I20240503 08:12:05.047118 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 50 FOLLOWER]: Advancing to term 51
I20240503 08:12:05.047137 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:05.047221 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14263, Committed index: 14263, Last appended: 50.14263, Last appended by leader: 14263, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.047363 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 50 FOLLOWER]: Advancing to term 51
I20240503 08:12:05.047894 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.048048 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 51 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.048271 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 51 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.048271 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 51 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.048380 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 51. Current term is 52.
I20240503 08:12:05.048591 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 51 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 51. Current term is 52.
I20240503 08:12:05.048631 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 51 election: Cancelling election due to peer responding with higher term
I20240503 08:12:05.048696 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 51 FOLLOWER]: Advancing to term 52
I20240503 08:12:05.049060 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 51.
I20240503 08:12:05.049294 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 52 FOLLOWER]: Leader election lost for term 51. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 51. Current term is 52.
I20240503 08:12:05.062193 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.062290 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 52 FOLLOWER]: Advancing to term 53
I20240503 08:12:05.062989 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.063135 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 53 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.063333 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 53 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.063395 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 52 FOLLOWER]: Advancing to term 53
I20240503 08:12:05.063350 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 53 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.063439 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 51 FOLLOWER]: Advancing to term 53
I20240503 08:12:05.063995 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 53.
I20240503 08:12:05.064064 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 53.
I20240503 08:12:05.064183 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:05.064280 26473 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 53 FOLLOWER]: Leader election won for term 53
I20240503 08:12:05.064335 26473 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 53 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.064392 26473 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14263, Committed index: 14263, Last appended: 50.14263, Last appended by leader: 14263, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.064843 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 50 to 53, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 53 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.088627 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 53 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 50 index: 14263. Preceding OpId from leader: term: 53 index: 14264. (index mismatch)
I20240503 08:12:05.088938 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14264, Last known committed idx: 14263, Time since last communication: 0.000s
I20240503 08:12:05.096206 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.096293 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 53 FOLLOWER]: Advancing to term 54
I20240503 08:12:05.096946 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.097093 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 54 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.097290 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 54 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.097295 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 54 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.097541 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.097652 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20240503 08:12:05.099586 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 53. Current term is 54. Ops: []
I20240503 08:12:05.099884 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14264, Last known committed idx: 14263, Time since last communication: 0.000s
I20240503 08:12:05.103229 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 53. Current term is 54. Ops: [53.14264-53.14264]
I20240503 08:12:05.103493 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 50.14263, Next index: 14264, Last known committed idx: 14263, Time since last communication: 0.000s
I20240503 08:12:05.120229 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 53 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:05.120353 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 53 FOLLOWER]: Advancing to term 54
I20240503 08:12:05.121042 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.121194 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 54 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.121430 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 54 candidate_status { last_received { term: 53 index: 14264 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.121430 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 54 candidate_status { last_received { term: 53 index: 14264 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.121562 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 54: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:05.121721 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.121855 26407 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20240503 08:12:05.122009 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 54 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 53. Current term is 54. Ops: []
I20240503 08:12:05.122190 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 53.14264, Next index: 14265, Last known committed idx: 14264, Time since last communication: 0.000s
I20240503 08:12:05.134318 26399 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 53 LEADER]: Stepping down as leader of term 53
I20240503 08:12:05.134402 26399 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 53 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.134454 26399 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14264, Committed index: 14264, Last appended: 53.14264, Last appended by leader: 14264, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.134584 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 53 FOLLOWER]: Advancing to term 54
I20240503 08:12:05.139941 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.140043 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 54 FOLLOWER]: Advancing to term 55
I20240503 08:12:05.140678 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.140834 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 55 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.141058 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 55 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.141122 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 54 FOLLOWER]: Advancing to term 55
I20240503 08:12:05.141088 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 55 candidate_status { last_received { term: 50 index: 14263 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.141188 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 54 FOLLOWER]: Advancing to term 55
I20240503 08:12:05.141709 26134 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 55 because replica has last-logged OpId of term: 53 index: 14264, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 14263.
I20240503 08:12:05.141726 26284 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 55 because replica has last-logged OpId of term: 53 index: 14264, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 14263.
I20240503 08:12:05.141933 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.142043 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20240503 08:12:05.157241 26407 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.157349 26407 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 55 FOLLOWER]: Advancing to term 56
I20240503 08:12:05.158020 26407 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.158186 26407 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 56 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.158390 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 56 candidate_status { last_received { term: 53 index: 14264 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.158402 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 56 candidate_status { last_received { term: 53 index: 14264 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.158466 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 55 FOLLOWER]: Advancing to term 56
I20240503 08:12:05.158502 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 55 FOLLOWER]: Advancing to term 56
I20240503 08:12:05.159044 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 56.
I20240503 08:12:05.159108 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 56.
I20240503 08:12:05.159255 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:05.159355 26407 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 56 FOLLOWER]: Leader election won for term 56
I20240503 08:12:05.159418 26407 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 56 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:05.159483 26407 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14264, Committed index: 14264, Last appended: 53.14264, Last appended by leader: 14264, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.159881 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 53 to 56, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 56 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.191354 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 56 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 53 index: 14264. Preceding OpId from leader: term: 56 index: 14265. (index mismatch)
I20240503 08:12:05.191648 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14265, Last known committed idx: 14264, Time since last communication: 0.000s
I20240503 08:12:05.195452 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 56 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 50 index: 14263. Preceding OpId from leader: term: 56 index: 14265. (index mismatch)
I20240503 08:12:05.195683 26407 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14265, Last known committed idx: 14263, Time since last communication: 0.000s
I20240503 08:12:05.252264 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 56 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:05.252355 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 56 FOLLOWER]: Advancing to term 57
I20240503 08:12:05.253026 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.253257 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 57 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.253345 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 57 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.253342 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 57 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.253628 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:05.253717 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20240503 08:12:05.257512 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 57 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 56. Current term is 57. Ops: []
I20240503 08:12:05.257797 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 56.14265, Next index: 14266, Last known committed idx: 14265, Time since last communication: 0.000s
I20240503 08:12:05.281049 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 56 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:05.281145 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 56 FOLLOWER]: Advancing to term 57
I20240503 08:12:05.281807 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.281955 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 57 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 56. Current term is 57. Ops: []
I20240503 08:12:05.281956 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 57 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.282104 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 57 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.282104 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 57 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.282223 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 57: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:05.282236 26407 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 56.14265, Next index: 14266, Last known committed idx: 14265, Time since last communication: 0.000s
I20240503 08:12:05.282380 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.282474 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20240503 08:12:05.284045 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 56 LEADER]: Stepping down as leader of term 56
I20240503 08:12:05.284107 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:05.284176 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14265, Committed index: 14265, Last appended: 56.14265, Last appended by leader: 14265, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.284217 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.284258 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 56 FOLLOWER]: Advancing to term 57
I20240503 08:12:05.284267 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 57 FOLLOWER]: Advancing to term 58
I20240503 08:12:05.284883 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.285001 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 56. Current term is 58. Ops: []
I20240503 08:12:05.285146 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 58 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.285166 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 58 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.285221 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 57 FOLLOWER]: Advancing to term 58
I20240503 08:12:05.285220 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 58 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.285401 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 57 FOLLOWER]: Advancing to term 58
I20240503 08:12:05.285840 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 58.
I20240503 08:12:05.285984 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 58.
I20240503 08:12:05.286073 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:05.286226 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 FOLLOWER]: Leader election won for term 58
I20240503 08:12:05.286288 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.286350 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14265, Committed index: 14265, Last appended: 56.14265, Last appended by leader: 14265, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.286845 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 56 to 58, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 58 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.318997 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.319110 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 58 FOLLOWER]: Advancing to term 59
I20240503 08:12:05.319790 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.319968 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 59 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.320187 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 59 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.320215 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 59 candidate_status { last_received { term: 56 index: 14265 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.320317 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 58 FOLLOWER]: Advancing to term 59
I20240503 08:12:05.320889 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 59.
I20240503 08:12:05.321097 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.321193 26384 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 FOLLOWER]: Leader election won for term 59
I20240503 08:12:05.321261 26384 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:05.321317 26384 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14265, Committed index: 14265, Last appended: 56.14265, Last appended by leader: 14265, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.321561 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 58. Current term is 59. Ops: []
I20240503 08:12:05.321755 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14266, Last known committed idx: 14265, Time since last communication: 0.000s
I20240503 08:12:05.321765 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 58 to 59, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 59 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.321847 26399 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 59.14266 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14265
I20240503 08:12:05.323020 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 59 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 58. Current term is 59. Ops: []
I20240503 08:12:05.323225 26399 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14266, Last known committed idx: 14265, Time since last communication: 0.000s
I20240503 08:12:05.340042 26473 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 LEADER]: Stepping down as leader of term 58
I20240503 08:12:05.340147 26473 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.340219 26473 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14265, Committed index: 14265, Last appended: 58.14266, Last appended by leader: 14266, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.340353 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 58 FOLLOWER]: Advancing to term 59
I20240503 08:12:05.346581 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 59 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 58 index: 14266. Preceding OpId from leader: term: 59 index: 14266. (term mismatch)
I20240503 08:12:05.346698 26284 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14265
I20240503 08:12:05.346712 26284 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted NO_OP operation due to leader change: 58.14266
I20240503 08:12:05.346727 26284 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 59 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:05.346989 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14266, Last known committed idx: 14265, Time since last communication: 0.000s
I20240503 08:12:05.347309 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 59 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 56 index: 14265. Preceding OpId from leader: term: 59 index: 14266. (index mismatch)
I20240503 08:12:05.347503 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14266, Last known committed idx: 14265, Time since last communication: 0.000s
I20240503 08:12:05.379074 26481 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 59 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:05.379185 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 59 FOLLOWER]: Advancing to term 60
I20240503 08:12:05.379593 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 59 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:05.379665 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 59 FOLLOWER]: Advancing to term 60
I20240503 08:12:05.380033 26481 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.380159 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 60 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 59. Current term is 60. Ops: []
I20240503 08:12:05.380190 26481 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 60 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.380235 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.380375 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 60 candidate_status { last_received { term: 59 index: 14266 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.380415 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 59.14266, Next index: 14267, Last known committed idx: 14266, Time since last communication: 0.000s
I20240503 08:12:05.380440 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 60 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.380400 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 60 candidate_status { last_received { term: 59 index: 14266 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.380445 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 60: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:05.380560 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 60 candidate_status { last_received { term: 59 index: 14266 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.380594 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 60 candidate_status { last_received { term: 59 index: 14266 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.380664 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 60: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:05.380779 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.380789 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:05.380928 26481 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20240503 08:12:05.380956 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20240503 08:12:05.383900 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 60 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 59. Current term is 60. Ops: []
I20240503 08:12:05.384140 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 59.14266, Next index: 14267, Last known committed idx: 14266, Time since last communication: 0.000s
I20240503 08:12:05.412621 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.412725 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 60 FOLLOWER]: Advancing to term 61
I20240503 08:12:05.413381 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.413547 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 61 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.413758 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 61 candidate_status { last_received { term: 59 index: 14266 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.413766 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 61 candidate_status { last_received { term: 59 index: 14266 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.413862 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 60 FOLLOWER]: Advancing to term 61
I20240503 08:12:05.414530 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 61.
I20240503 08:12:05.414690 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 61 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 59. Current term is 61. Ops: []
I20240503 08:12:05.414716 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:05.414731 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 59. Current term is 61. Ops: []
I20240503 08:12:05.414791 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 FOLLOWER]: Leader election won for term 61
I20240503 08:12:05.414844 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:05.414892 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14266, Committed index: 14266, Last appended: 59.14266, Last appended by leader: 14266, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.415064 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 59.14266, Next index: 14267, Last known committed idx: 14266, Time since last communication: 0.000s
I20240503 08:12:05.415169 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 59.14266, Next index: 14267, Last known committed idx: 14266, Time since last communication: 0.000s
I20240503 08:12:05.415288 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 59 to 61, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 61 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.433574 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 LEADER]: Stepping down as leader of term 59
I20240503 08:12:05.433671 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:05.433768 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14266, Committed index: 14266, Last appended: 59.14266, Last appended by leader: 14266, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.433878 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 59 FOLLOWER]: Advancing to term 60
I20240503 08:12:05.439857 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 61 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 59 index: 14266. Preceding OpId from leader: term: 61 index: 14267. (index mismatch)
I20240503 08:12:05.440155 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14267, Last known committed idx: 14266, Time since last communication: 0.000s
I20240503 08:12:05.441583 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 60 FOLLOWER]: Advancing to term 61
I20240503 08:12:05.442241 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 61 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 59 index: 14266. Preceding OpId from leader: term: 61 index: 14267. (index mismatch)
I20240503 08:12:05.442466 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14267, Last known committed idx: 14266, Time since last communication: 0.000s
I20240503 08:12:05.472184 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 61 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:05.472275 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 61 FOLLOWER]: Advancing to term 62
I20240503 08:12:05.472930 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.473093 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 62 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.473263 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 62 candidate_status { last_received { term: 61 index: 14267 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.473263 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 62 candidate_status { last_received { term: 61 index: 14267 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.473543 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:05.473632 26473 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20240503 08:12:05.476191 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 62 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 61. Current term is 62. Ops: []
I20240503 08:12:05.476410 26494 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 61.14267, Next index: 14268, Last known committed idx: 14267, Time since last communication: 0.000s
I20240503 08:12:05.496870 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 61 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:05.496963 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 61 FOLLOWER]: Advancing to term 62
I20240503 08:12:05.497633 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.497776 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 62 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.497939 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 62 candidate_status { last_received { term: 61 index: 14267 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.497954 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 62 candidate_status { last_received { term: 61 index: 14267 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.498093 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 62: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:05.498237 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.498330 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20240503 08:12:05.499650 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 LEADER]: Stepping down as leader of term 61
I20240503 08:12:05.499699 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:05.499765 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14267, Committed index: 14267, Last appended: 61.14267, Last appended by leader: 14267, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.499858 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 61 FOLLOWER]: Advancing to term 62
I20240503 08:12:05.504419 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 62 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.504487 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 62 FOLLOWER]: Advancing to term 63
I20240503 08:12:05.505158 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.505304 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 63 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.505503 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 63 candidate_status { last_received { term: 61 index: 14267 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.505512 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 63 candidate_status { last_received { term: 61 index: 14267 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.505591 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 62 FOLLOWER]: Advancing to term 63
I20240503 08:12:05.505596 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 62 FOLLOWER]: Advancing to term 63
I20240503 08:12:05.506230 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 63.
I20240503 08:12:05.506230 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 63.
I20240503 08:12:05.506448 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:05.506563 26473 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 63 FOLLOWER]: Leader election won for term 63
I20240503 08:12:05.506649 26473 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 63 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.506789 26473 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14267, Committed index: 14267, Last appended: 61.14267, Last appended by leader: 14267, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.507277 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 61 to 63, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 63 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.533850 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 63 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 61 index: 14267. Preceding OpId from leader: term: 63 index: 14268. (index mismatch)
I20240503 08:12:05.534163 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14268, Last known committed idx: 14267, Time since last communication: 0.000s
I20240503 08:12:05.534842 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 63 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 61 index: 14267. Preceding OpId from leader: term: 63 index: 14268. (index mismatch)
I20240503 08:12:05.535034 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14268, Last known committed idx: 14267, Time since last communication: 0.000s
I20240503 08:12:05.591615 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 63 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:05.591707 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 63 FOLLOWER]: Advancing to term 64
I20240503 08:12:05.592353 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.592547 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 64 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.592795 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 64 candidate_status { last_received { term: 63 index: 14268 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.592795 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 64 candidate_status { last_received { term: 63 index: 14268 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.593075 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.593170 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20240503 08:12:05.593742 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 63. Current term is 64. Ops: []
I20240503 08:12:05.593950 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 63.14268, Next index: 14269, Last known committed idx: 14268, Time since last communication: 0.000s
I20240503 08:12:05.614473 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 63 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:05.614571 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 63 FOLLOWER]: Advancing to term 64
I20240503 08:12:05.615239 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.615430 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 64 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.615664 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 64 candidate_status { last_received { term: 63 index: 14268 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.615796 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 64: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:05.615664 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 64 candidate_status { last_received { term: 63 index: 14268 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.615993 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.616125 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20240503 08:12:05.617786 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 63. Current term is 64. Ops: []
I20240503 08:12:05.617954 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 64 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 63. Current term is 64. Ops: []
I20240503 08:12:05.618010 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 63.14268, Next index: 14269, Last known committed idx: 14268, Time since last communication: 0.000s
I20240503 08:12:05.618152 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 63.14268, Next index: 14269, Last known committed idx: 14268, Time since last communication: 0.000s
I20240503 08:12:05.624591 26399 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 63 LEADER]: Stepping down as leader of term 63
I20240503 08:12:05.624666 26399 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 63 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.624722 26399 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14268, Committed index: 14268, Last appended: 63.14268, Last appended by leader: 14268, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.624835 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 63 FOLLOWER]: Advancing to term 64
I20240503 08:12:05.637782 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.637885 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 64 FOLLOWER]: Advancing to term 65
I20240503 08:12:05.638592 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.638783 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 65 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.639016 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 65 candidate_status { last_received { term: 63 index: 14268 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.639098 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 64 FOLLOWER]: Advancing to term 65
I20240503 08:12:05.639047 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 65 candidate_status { last_received { term: 63 index: 14268 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.639149 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 64 FOLLOWER]: Advancing to term 65
I20240503 08:12:05.639678 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 65.
I20240503 08:12:05.639809 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 65.
I20240503 08:12:05.639912 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:05.639999 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 65 FOLLOWER]: Leader election won for term 65
I20240503 08:12:05.640050 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 65 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:05.640110 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14268, Committed index: 14268, Last appended: 63.14268, Last appended by leader: 14268, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.640770 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 63 to 65, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 65 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.666044 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 65 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 63 index: 14268. Preceding OpId from leader: term: 65 index: 14269. (index mismatch)
I20240503 08:12:05.666339 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14269, Last known committed idx: 14268, Time since last communication: 0.000s
I20240503 08:12:05.667497 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 65 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 63 index: 14268. Preceding OpId from leader: term: 65 index: 14269. (index mismatch)
I20240503 08:12:05.667716 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14269, Last known committed idx: 14268, Time since last communication: 0.000s
I20240503 08:12:05.698957 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 65 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:05.699052 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 65 FOLLOWER]: Advancing to term 66
I20240503 08:12:05.699692 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.699842 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 66 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.700049 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 66 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.700037 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 66 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.700314 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:05.700402 26399 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20240503 08:12:05.703284 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 65. Current term is 66. Ops: []
I20240503 08:12:05.703531 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 65.14269, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.725965 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 65 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:05.726079 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 65 FOLLOWER]: Advancing to term 66
I20240503 08:12:05.726904 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.727123 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 66 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.727380 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 66 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.727380 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 66 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.727581 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 66: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:05.727633 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 65. Current term is 66. Ops: []
I20240503 08:12:05.727752 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.727844 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20240503 08:12:05.727856 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 65.14269, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.730794 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 66 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 65. Current term is 66. Ops: []
I20240503 08:12:05.731001 26399 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.731029 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 65.14269, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.731068 26399 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 66 FOLLOWER]: Advancing to term 67
I20240503 08:12:05.731701 26399 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.731824 26399 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 67 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.732008 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 67 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.732026 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 67 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.732084 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 66 FOLLOWER]: Advancing to term 67
I20240503 08:12:05.732630 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 67.
I20240503 08:12:05.732815 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:05.732908 26399 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 67 FOLLOWER]: Leader election won for term 67
I20240503 08:12:05.732970 26399 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 67 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.733028 26399 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14269, Committed index: 14269, Last appended: 65.14269, Last appended by leader: 14269, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.733489 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 65 to 67, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 67 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:05.762213 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 67 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 65. Current term is 67. Ops: []
I20240503 08:12:05.762501 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 65.14269, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.762802 26210 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 65 LEADER]: Stepping down as leader of term 65
I20240503 08:12:05.762845 26210 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 65 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:05.762903 26210 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14269, Committed index: 14269, Last appended: 65.14269, Last appended by leader: 14269, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.763001 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 65 FOLLOWER]: Advancing to term 67
I20240503 08:12:05.763643 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 67 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 65 index: 14269. Preceding OpId from leader: term: 67 index: 14270. (index mismatch)
I20240503 08:12:05.763880 26399 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.764506 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 67 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.764555 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 67 FOLLOWER]: Advancing to term 68
I20240503 08:12:05.765304 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.765419 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 68 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.765568 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 68 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.765573 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 68 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.765755 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.765820 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I20240503 08:12:05.768397 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 67. Current term is 68. Ops: []
I20240503 08:12:05.768625 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.776055 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 67. Current term is 68. Ops: [67.14270-67.14270]
I20240503 08:12:05.776412 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 65.14269, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
W20240503 08:12:05.801194 26349 lock_manager.cc:494] Txn 4 has not acquired the partition lock after 1000ms
I20240503 08:12:05.808449 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 67. Current term is 68. Ops: [67.14270-67.14270]
I20240503 08:12:05.808746 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 65.14269, Next index: 14270, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.823930 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.824056 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 68 FOLLOWER]: Advancing to term 69
I20240503 08:12:05.824774 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.824966 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 69 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.825244 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 69 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.825244 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 69 candidate_status { last_received { term: 65 index: 14269 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.825528 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.825661 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I20240503 08:12:05.842178 26473 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 67 LEADER]: Stepping down as leader of term 67
I20240503 08:12:05.842293 26473 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 67 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:05.842360 26473 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14270, Committed index: 14270, Last appended: 67.14270, Last appended by leader: 14270, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.842497 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 67 FOLLOWER]: Advancing to term 68
I20240503 08:12:05.845291 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 67 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:05.845388 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 67 FOLLOWER]: Advancing to term 68
I20240503 08:12:05.846105 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.846277 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 68 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.846534 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 68 candidate_status { last_received { term: 67 index: 14270 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.846534 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 68 candidate_status { last_received { term: 67 index: 14270 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.846658 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 68. Current term is 69.
I20240503 08:12:05.846866 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 68 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 68. Current term is 69.
I20240503 08:12:05.846910 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 68 election: Cancelling election due to peer responding with higher term
I20240503 08:12:05.847016 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 68 FOLLOWER]: Advancing to term 69
I20240503 08:12:05.847399 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 68.
I20240503 08:12:05.847754 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 69 FOLLOWER]: Leader election lost for term 68. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 69 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 68. Current term is 69.
I20240503 08:12:05.888559 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.888669 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 69 FOLLOWER]: Advancing to term 70
I20240503 08:12:05.889308 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.889480 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 70 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:05.889703 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 70 candidate_status { last_received { term: 67 index: 14270 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:05.889777 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 69 FOLLOWER]: Advancing to term 70
I20240503 08:12:05.889752 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 70 candidate_status { last_received { term: 67 index: 14270 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.889850 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 68 FOLLOWER]: Advancing to term 70
I20240503 08:12:05.890372 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 70.
I20240503 08:12:05.890508 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 70.
I20240503 08:12:05.890656 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:05.890759 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 70 FOLLOWER]: Leader election won for term 70
I20240503 08:12:05.890836 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 70 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:05.890889 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14270, Committed index: 14270, Last appended: 67.14270, Last appended by leader: 14270, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.891461 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 67 to 70, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 70 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:05.913652 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 70 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 65 index: 14269. Preceding OpId from leader: term: 70 index: 14271. (index mismatch)
I20240503 08:12:05.913985 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14271, Last known committed idx: 14269, Time since last communication: 0.000s
I20240503 08:12:05.917647 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 70 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 67 index: 14270. Preceding OpId from leader: term: 70 index: 14271. (index mismatch)
I20240503 08:12:05.917951 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14271, Last known committed idx: 14270, Time since last communication: 0.000s
I20240503 08:12:05.945147 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 70 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:05.945242 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 70 FOLLOWER]: Advancing to term 71
I20240503 08:12:05.945946 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.946103 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 71 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.946301 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 71 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.946308 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 71 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.946696 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 71 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.946877 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 71 FOLLOWER]: Leader election lost for term 71. Reason: could not achieve majority
I20240503 08:12:05.952275 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 71 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 70. Current term is 71. Ops: []
I20240503 08:12:05.952602 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 70.14271, Next index: 14272, Last known committed idx: 14271, Time since last communication: 0.000s
I20240503 08:12:05.982738 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 71 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 70. Current term is 71. Ops: []
I20240503 08:12:05.982997 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 70.14271, Next index: 14272, Last known committed idx: 14271, Time since last communication: 0.000s
I20240503 08:12:05.991509 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 71 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:05.991598 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 71 FOLLOWER]: Advancing to term 72
I20240503 08:12:05.992278 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:05.992429 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 72 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:05.992643 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 72 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:05.992664 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 72 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:05.992884 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 72 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:05.992982 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Leader election lost for term 72. Reason: could not achieve majority
I20240503 08:12:06.018532 26384 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 70 LEADER]: Stepping down as leader of term 70
I20240503 08:12:06.018666 26384 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 70 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:06.018746 26384 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14271, Committed index: 14271, Last appended: 70.14271, Last appended by leader: 14271, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.018882 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 70 FOLLOWER]: Advancing to term 71
I20240503 08:12:06.023864 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 70 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:06.023945 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 70 FOLLOWER]: Advancing to term 71
I20240503 08:12:06.024551 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.024708 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 71 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.024922 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 71 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.024924 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 71 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.025025 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 71. Current term is 72.
I20240503 08:12:06.025177 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 71 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 71. Current term is 72.
I20240503 08:12:06.025223 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 71 election: Cancelling election due to peer responding with higher term
I20240503 08:12:06.025306 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 71 FOLLOWER]: Advancing to term 72
I20240503 08:12:06.025656 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 71.
I20240503 08:12:06.025825 26473 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 72 FOLLOWER]: Leader election lost for term 71. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 71. Current term is 72.
I20240503 08:12:06.048934 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.049057 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 72 FOLLOWER]: Advancing to term 73
I20240503 08:12:06.049806 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.050004 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 73 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.050240 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 73 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.050264 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 73 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.050349 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 72 FOLLOWER]: Advancing to term 73
I20240503 08:12:06.050361 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 71 FOLLOWER]: Advancing to term 73
I20240503 08:12:06.050994 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 73.
I20240503 08:12:06.051002 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 73.
I20240503 08:12:06.051219 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:06.051330 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 73 FOLLOWER]: Leader election won for term 73
I20240503 08:12:06.051472 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 73 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:06.051524 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14271, Committed index: 14271, Last appended: 70.14271, Last appended by leader: 14271, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.052022 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 70 to 73, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 73 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.082731 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.082835 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 73 FOLLOWER]: Advancing to term 74
I20240503 08:12:06.083578 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.083745 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 74 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.083987 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 74 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.084012 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 74 candidate_status { last_received { term: 70 index: 14271 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.084093 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 73 FOLLOWER]: Advancing to term 74
I20240503 08:12:06.084736 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 74.
I20240503 08:12:06.084956 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:06.085062 26384 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 FOLLOWER]: Leader election won for term 74
I20240503 08:12:06.085120 26384 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:06.085175 26384 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14271, Committed index: 14271, Last appended: 70.14271, Last appended by leader: 14271, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.088409 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 73. Current term is 74. Ops: []
I20240503 08:12:06.088409 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 74 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 73. Current term is 74. Ops: []
I20240503 08:12:06.088704 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14272, Last known committed idx: 14271, Time since last communication: 0.000s
I20240503 08:12:06.088860 26502 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 74.14272 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14271
I20240503 08:12:06.088941 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14272, Last known committed idx: 14271, Time since last communication: 0.000s
I20240503 08:12:06.096930 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 73 to 74, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 74 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:06.111017 26516 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 73 LEADER]: Stepping down as leader of term 73
I20240503 08:12:06.111110 26516 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 73 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:06.111192 26516 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14271, Committed index: 14271, Last appended: 73.14272, Last appended by leader: 14272, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.111317 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 73 FOLLOWER]: Advancing to term 74
I20240503 08:12:06.115090 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 74 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 73 index: 14272. Preceding OpId from leader: term: 74 index: 14272. (term mismatch)
I20240503 08:12:06.115159 26134 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14271
I20240503 08:12:06.115181 26134 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 73.14272
I20240503 08:12:06.115203 26134 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 74 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:06.115433 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14272, Last known committed idx: 14271, Time since last communication: 0.000s
I20240503 08:12:06.120658 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 74 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 70 index: 14271. Preceding OpId from leader: term: 74 index: 14272. (index mismatch)
I20240503 08:12:06.120911 26384 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14272, Last known committed idx: 14271, Time since last communication: 0.000s
I20240503 08:12:06.203632 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 74 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:06.203748 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 74 FOLLOWER]: Advancing to term 75
I20240503 08:12:06.204586 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.204739 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 75 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.204881 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 75 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.204881 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 75 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.205128 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.205226 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20240503 08:12:06.208321 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 75 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 74. Current term is 75. Ops: []
I20240503 08:12:06.208568 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 74.14272, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.234366 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 75 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 74. Current term is 75. Ops: []
I20240503 08:12:06.234735 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 74.14272, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.247961 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 75 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.248075 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 75 FOLLOWER]: Advancing to term 76
I20240503 08:12:06.248740 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.248785 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 LEADER]: Stepping down as leader of term 74
I20240503 08:12:06.248854 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:06.248942 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14272, Committed index: 14272, Last appended: 74.14272, Last appended by leader: 14272, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.249064 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 74 FOLLOWER]: Advancing to term 75
I20240503 08:12:06.249085 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 76 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.249128 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 76 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.249238 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 76 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.249797 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 75 FOLLOWER]: Advancing to term 76
I20240503 08:12:06.250495 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 76.
I20240503 08:12:06.250761 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 76 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.250941 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 76 FOLLOWER]: Leader election won for term 76
I20240503 08:12:06.251011 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 76 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:06.251076 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14272, Committed index: 14272, Last appended: 74.14272, Last appended by leader: 14272, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.251645 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 74 to 76, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 76 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.258417 26481 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 74 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:06.258520 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 74 FOLLOWER]: Advancing to term 75
I20240503 08:12:06.259191 26481 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.259341 26481 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 75 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.259570 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 75 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.259596 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 75 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.259665 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 75. Current term is 76.
I20240503 08:12:06.259797 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 75 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 75. Current term is 76.
I20240503 08:12:06.259837 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 75 election: Cancelling election due to peer responding with higher term
I20240503 08:12:06.259899 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 75 FOLLOWER]: Advancing to term 76
I20240503 08:12:06.260438 26481 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 76 FOLLOWER]: Leader election lost for term 75. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 76 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 75. Current term is 76.
I20240503 08:12:06.282322 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 76 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 74 index: 14272. Preceding OpId from leader: term: 76 index: 14273. (index mismatch)
I20240503 08:12:06.282657 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.283098 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 76 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.283154 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 76 FOLLOWER]: Advancing to term 77
I20240503 08:12:06.283792 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.284066 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 77 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.284243 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 77 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.284243 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 77 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.284530 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 77 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.284622 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Leader election lost for term 77. Reason: could not achieve majority
I20240503 08:12:06.284822 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 76. Current term is 77. Ops: []
I20240503 08:12:06.284993 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.309806 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 76. Current term is 77. Ops: [76.14273-76.14273]
I20240503 08:12:06.310065 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 74.14272, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.315420 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 76. Current term is 77. Ops: [76.14273-76.14273]
I20240503 08:12:06.315721 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 74.14272, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.321192 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.321281 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 77 FOLLOWER]: Advancing to term 78
I20240503 08:12:06.321933 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.322095 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 78 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.322299 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 78 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.322299 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 78 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.322568 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 78 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.322690 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 78 FOLLOWER]: Leader election lost for term 78. Reason: could not achieve majority
I20240503 08:12:06.348577 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 78 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 76. Current term is 78. Ops: [76.14273-76.14273]
I20240503 08:12:06.349020 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 74.14272, Next index: 14273, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.359362 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 78 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.359463 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 78 FOLLOWER]: Advancing to term 79
I20240503 08:12:06.360149 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.360314 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 79 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.360538 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 79 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.360574 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 79 candidate_status { last_received { term: 74 index: 14272 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.360780 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.360877 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20240503 08:12:06.363346 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 76 LEADER]: Stepping down as leader of term 76
I20240503 08:12:06.363395 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 76 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:06.363448 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14273, Committed index: 14273, Last appended: 76.14273, Last appended by leader: 14273, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.363551 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 76 FOLLOWER]: Advancing to term 77
I20240503 08:12:06.364466 26481 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 76 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:06.364508 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 76 FOLLOWER]: Advancing to term 77
I20240503 08:12:06.365164 26481 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.365366 26481 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 77 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.365520 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 77 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.365520 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 77 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.365626 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 77. Current term is 79.
I20240503 08:12:06.365834 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 77 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 77. Current term is 79.
I20240503 08:12:06.365898 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 77 election: Cancelling election due to peer responding with higher term
I20240503 08:12:06.365957 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 77 FOLLOWER]: Advancing to term 79
I20240503 08:12:06.366425 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 77 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 77.
I20240503 08:12:06.366586 26481 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 79 FOLLOWER]: Leader election lost for term 77. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 77. Current term is 79.
I20240503 08:12:06.402343 26481 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.402454 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 79 FOLLOWER]: Advancing to term 80
I20240503 08:12:06.403162 26481 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.403328 26481 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 80 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.403543 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 80 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.403529 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 80 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.403625 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 77 FOLLOWER]: Advancing to term 80
I20240503 08:12:06.403631 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 79 FOLLOWER]: Advancing to term 80
I20240503 08:12:06.404307 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 80.
I20240503 08:12:06.404309 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 80.
I20240503 08:12:06.404532 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:06.404623 26481 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 80 FOLLOWER]: Leader election won for term 80
I20240503 08:12:06.404672 26481 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 80 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:06.404731 26481 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14273, Committed index: 14273, Last appended: 76.14273, Last appended by leader: 14273, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.405200 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 76 to 80, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 80 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.428745 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 80 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 74 index: 14272. Preceding OpId from leader: term: 80 index: 14274. (index mismatch)
I20240503 08:12:06.429080 26481 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14274, Last known committed idx: 14272, Time since last communication: 0.000s
I20240503 08:12:06.441514 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 80 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.441628 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 80 FOLLOWER]: Advancing to term 81
I20240503 08:12:06.442478 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.442642 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 80. Current term is 81. Ops: []
I20240503 08:12:06.442685 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 81 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.442873 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 81 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.442896 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14274, Last known committed idx: 14273, Time since last communication: 0.000s
I20240503 08:12:06.442920 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 81 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.443109 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.443202 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20240503 08:12:06.450587 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 80. Current term is 81. Ops: [80.14274-80.14274]
I20240503 08:12:06.450891 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 76.14273, Next index: 14274, Last known committed idx: 14273, Time since last communication: 0.000s
I20240503 08:12:06.461277 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 80 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:06.461386 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 80 FOLLOWER]: Advancing to term 81
I20240503 08:12:06.462051 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.462231 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 81 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.462483 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 81 candidate_status { last_received { term: 80 index: 14274 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.462499 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 81 candidate_status { last_received { term: 80 index: 14274 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.462592 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 81: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:06.462822 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.462980 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20240503 08:12:06.467083 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 81 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 80. Current term is 81. Ops: []
I20240503 08:12:06.467366 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 80.14274, Next index: 14275, Last known committed idx: 14274, Time since last communication: 0.000s
I20240503 08:12:06.482781 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.482877 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 81 FOLLOWER]: Advancing to term 82
I20240503 08:12:06.483561 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.483700 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 82 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 80. Current term is 82. Ops: [80.14274-80.14274]
I20240503 08:12:06.483920 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 82 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.483704 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 82 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.483888 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 82 candidate_status { last_received { term: 76 index: 14273 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.484057 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 76.14273, Next index: 14274, Last known committed idx: 14273, Time since last communication: 0.000s
I20240503 08:12:06.484056 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 81 FOLLOWER]: Advancing to term 82
I20240503 08:12:06.484772 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 82 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 82 because replica has last-logged OpId of term: 80 index: 14274, which is greater than that of the candidate, which has last-logged OpId of term: 76 index: 14273.
I20240503 08:12:06.484968 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 82 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.485065 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 82 FOLLOWER]: Leader election lost for term 82. Reason: could not achieve majority
I20240503 08:12:06.496227 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 82 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 80. Current term is 82. Ops: []
I20240503 08:12:06.496543 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 80.14274, Next index: 14275, Last known committed idx: 14274, Time since last communication: 0.000s
I20240503 08:12:06.506762 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 82 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.506878 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 82 FOLLOWER]: Advancing to term 83
I20240503 08:12:06.507601 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.507848 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 83 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.508062 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 83 candidate_status { last_received { term: 80 index: 14274 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.508062 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 83 candidate_status { last_received { term: 80 index: 14274 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.508189 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 82 FOLLOWER]: Advancing to term 83
I20240503 08:12:06.508812 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 83.
I20240503 08:12:06.509002 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.509097 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 83 FOLLOWER]: Leader election won for term 83
I20240503 08:12:06.509158 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 83 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:06.509214 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14274, Committed index: 14274, Last appended: 80.14274, Last appended by leader: 14274, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.509675 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 80 to 83, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 83 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:06.510293 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 83 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 80. Current term is 83. Ops: [80.14274-80.14274]
I20240503 08:12:06.510601 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 76.14273, Next index: 14274, Last known committed idx: 14273, Time since last communication: 0.000s
I20240503 08:12:06.515622 26481 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 80 LEADER]: Stepping down as leader of term 80
I20240503 08:12:06.515699 26481 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 80 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:06.515769 26481 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14274, Committed index: 14274, Last appended: 80.14274, Last appended by leader: 14274, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.515866 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 80 FOLLOWER]: Advancing to term 81
I20240503 08:12:06.531921 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 81 FOLLOWER]: Advancing to term 83
I20240503 08:12:06.532639 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 83 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 80 index: 14274. Preceding OpId from leader: term: 83 index: 14275. (index mismatch)
I20240503 08:12:06.532905 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14275, Last known committed idx: 14274, Time since last communication: 0.000s
I20240503 08:12:06.540046 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 83 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 76 index: 14273. Preceding OpId from leader: term: 83 index: 14275. (index mismatch)
I20240503 08:12:06.540313 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14275, Last known committed idx: 14273, Time since last communication: 0.000s
I20240503 08:12:06.564751 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 83 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:06.564852 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 83 FOLLOWER]: Advancing to term 84
I20240503 08:12:06.565510 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.565665 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 84 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.565888 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 84 candidate_status { last_received { term: 83 index: 14275 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.565888 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 84 candidate_status { last_received { term: 83 index: 14275 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.566130 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 84 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:06.566221 26473 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Leader election lost for term 84. Reason: could not achieve majority
I20240503 08:12:06.569900 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 83. Current term is 84. Ops: []
I20240503 08:12:06.570127 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 83.14275, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
I20240503 08:12:06.571098 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 83 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:06.571166 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 83 FOLLOWER]: Advancing to term 84
I20240503 08:12:06.571951 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.572098 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 84 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 83. Current term is 84. Ops: []
I20240503 08:12:06.572145 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 84 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.572268 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 84 candidate_status { last_received { term: 83 index: 14275 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.572376 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 84 candidate_status { last_received { term: 83 index: 14275 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.572444 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 84: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:06.572537 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 83.14275, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
I20240503 08:12:06.572597 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 84 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.572659 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 84 FOLLOWER]: Leader election lost for term 84. Reason: could not achieve majority
I20240503 08:12:06.593991 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 84 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 83. Current term is 84. Ops: []
I20240503 08:12:06.594254 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 83.14275, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
I20240503 08:12:06.601882 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 83. Current term is 84. Ops: []
I20240503 08:12:06.602228 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 83.14275, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
I20240503 08:12:06.602313 26473 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.602353 26473 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 84 FOLLOWER]: Advancing to term 85
I20240503 08:12:06.603034 26473 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.603164 26473 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 85 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.603353 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 85 candidate_status { last_received { term: 83 index: 14275 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.603353 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 85 candidate_status { last_received { term: 83 index: 14275 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.603459 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 84 FOLLOWER]: Advancing to term 85
I20240503 08:12:06.604068 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 85.
I20240503 08:12:06.604240 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:06.604331 26473 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 85 FOLLOWER]: Leader election won for term 85
I20240503 08:12:06.604399 26473 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 85 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:06.604454 26473 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14275, Committed index: 14275, Last appended: 83.14275, Last appended by leader: 14275, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.605038 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 83 to 85, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 85 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:06.606824 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 82 current term: 85 is higher.
I20240503 08:12:06.617090 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 85 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 83. Current term is 85. Ops: []
I20240503 08:12:06.617458 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 83.14275, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
I20240503 08:12:06.626078 26384 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 83 LEADER]: Stepping down as leader of term 83
I20240503 08:12:06.626173 26384 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 83 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:06.626250 26384 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14275, Committed index: 14275, Last appended: 83.14275, Last appended by leader: 14275, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.626374 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 83 FOLLOWER]: Advancing to term 84
I20240503 08:12:06.628264 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 84 FOLLOWER]: Advancing to term 85
I20240503 08:12:06.628877 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 85 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 83 index: 14275. Preceding OpId from leader: term: 85 index: 14276. (index mismatch)
I20240503 08:12:06.629174 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
I20240503 08:12:06.630219 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 85 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 83 index: 14275. Preceding OpId from leader: term: 85 index: 14276. (index mismatch)
I20240503 08:12:06.630429 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14276, Last known committed idx: 14275, Time since last communication: 0.000s
W20240503 08:12:06.642179 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 82 current term: 85 is higher.
W20240503 08:12:06.654400 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 83 current term: 85 is higher.
I20240503 08:12:06.660909 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 85 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:06.661015 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 85 FOLLOWER]: Advancing to term 86
I20240503 08:12:06.661697 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.661828 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 85 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:06.661901 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 85 FOLLOWER]: Advancing to term 86
I20240503 08:12:06.661975 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 86 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.662077 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 86 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.662075 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 86 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.662858 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.662972 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 86: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:06.663086 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 86 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.663166 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 86 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.663199 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 86 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.663206 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 86 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.663267 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Leader election lost for term 86. Reason: could not achieve majority
I20240503 08:12:06.663319 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 86: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:06.663486 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 86 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.663594 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Leader election lost for term 86. Reason: could not achieve majority
I20240503 08:12:06.665738 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 85. Current term is 86. Ops: []
I20240503 08:12:06.665793 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 85. Current term is 86. Ops: []
I20240503 08:12:06.665947 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 85.14276, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.666038 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 85.14276, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.690929 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 85. Current term is 86. Ops: []
I20240503 08:12:06.691231 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 85.14276, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.698675 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 85. Current term is 86. Ops: []
I20240503 08:12:06.698968 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 85.14276, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.704077 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.704161 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 86 FOLLOWER]: Advancing to term 87
I20240503 08:12:06.704427 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.704494 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 86 FOLLOWER]: Advancing to term 87
I20240503 08:12:06.704919 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.705062 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.705230 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 87 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.705299 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 87 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.705262 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 87 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.705343 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 87: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:06.705272 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 87 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.705440 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 87 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.705513 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 87 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.705574 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 87: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:06.705575 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 87 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.705688 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 87 FOLLOWER]: Leader election lost for term 87. Reason: could not achieve majority
I20240503 08:12:06.705767 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 87 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.705859 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 87 FOLLOWER]: Leader election lost for term 87. Reason: could not achieve majority
I20240503 08:12:06.723863 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 87 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 85. Current term is 87. Ops: []
I20240503 08:12:06.724143 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 85.14276, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.726238 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 87 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 85. Current term is 87. Ops: []
I20240503 08:12:06.726450 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 85.14276, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.743960 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 87 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.744072 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 87 FOLLOWER]: Advancing to term 88
I20240503 08:12:06.744844 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.745005 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 88 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.745229 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 88 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.745360 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 87 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.745395 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 87 FOLLOWER]: Advancing to term 88
I20240503 08:12:06.745978 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.745229 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 88 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.746106 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 88 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.746131 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 88: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:06.746276 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 88 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.746285 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 88 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.746232 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 88 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.746430 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Leader election lost for term 88. Reason: could not achieve majority
I20240503 08:12:06.746496 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 88: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:06.746673 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 88 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.746762 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Leader election lost for term 88. Reason: could not achieve majority
I20240503 08:12:06.754127 26481 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 85 LEADER]: Stepping down as leader of term 85
I20240503 08:12:06.754202 26481 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 85 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:06.754276 26481 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14276, Committed index: 14276, Last appended: 85.14276, Last appended by leader: 14276, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.754392 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 85 FOLLOWER]: Advancing to term 86
I20240503 08:12:06.780799 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 86 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.780910 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 86 FOLLOWER]: Advancing to term 87
I20240503 08:12:06.781713 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.781884 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 87 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.782121 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 87 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.782191 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 87. Current term is 88.
I20240503 08:12:06.782127 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 87 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.782248 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 87. Current term is 88.
I20240503 08:12:06.782342 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 87 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 87. Current term is 88.
I20240503 08:12:06.782426 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 87 election: Cancelling election due to peer responding with higher term
I20240503 08:12:06.782512 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 87 FOLLOWER]: Advancing to term 88
I20240503 08:12:06.782531 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 87 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 87. Current term is 88.
I20240503 08:12:06.783119 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 88 FOLLOWER]: Leader election lost for term 87. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 87. Current term is 88.
I20240503 08:12:06.793242 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.793351 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 88 FOLLOWER]: Advancing to term 89
I20240503 08:12:06.794032 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.794188 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 89 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.794422 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 89 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.794485 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 88 FOLLOWER]: Advancing to term 89
I20240503 08:12:06.794441 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 89 candidate_status { last_received { term: 85 index: 14276 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.794550 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 88 FOLLOWER]: Advancing to term 89
I20240503 08:12:06.795089 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 89.
I20240503 08:12:06.795148 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 89 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 89.
I20240503 08:12:06.795313 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 89 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:06.795415 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 89 FOLLOWER]: Leader election won for term 89
I20240503 08:12:06.795482 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 89 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:06.795542 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14276, Committed index: 14276, Last appended: 85.14276, Last appended by leader: 14276, Current term: 89, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.796003 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 85 to 89, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 89 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:06.802208 26351 lock_manager.cc:494] Txn 4 has not acquired the partition lock after 1000ms
I20240503 08:12:06.803092 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 89 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 85 index: 14276. Preceding OpId from leader: term: 89 index: 14278. (index mismatch)
I20240503 08:12:06.803120 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 89 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 85 index: 14276. Preceding OpId from leader: term: 89 index: 14278. (index mismatch)
I20240503 08:12:06.803367 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.803507 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14277, Last known committed idx: 14276, Time since last communication: 0.000s
I20240503 08:12:06.835037 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 89 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:06.835052 26384 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 89 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:06.835186 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 89 FOLLOWER]: Advancing to term 90
I20240503 08:12:06.835192 26384 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 89 FOLLOWER]: Advancing to term 90
I20240503 08:12:06.836047 26384 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.836047 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.836306 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 90 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.836306 26384 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 90 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.836488 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 90 candidate_status { last_received { term: 89 index: 14278 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.836545 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 90 candidate_status { last_received { term: 89 index: 14278 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.836560 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 90: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:06.836506 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 90 candidate_status { last_received { term: 89 index: 14278 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.836571 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 90 candidate_status { last_received { term: 89 index: 14278 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.836642 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 90: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:06.836792 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 90 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:06.836792 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 90 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.836920 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Leader election lost for term 90. Reason: could not achieve majority
I20240503 08:12:06.836921 26384 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 90 FOLLOWER]: Leader election lost for term 90. Reason: could not achieve majority
I20240503 08:12:06.838657 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 90 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 89. Current term is 90. Ops: []
I20240503 08:12:06.838928 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 89.14278, Next index: 14279, Last known committed idx: 14278, Time since last communication: 0.000s
I20240503 08:12:06.839532 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 89. Current term is 90. Ops: []
I20240503 08:12:06.839812 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 89.14278, Next index: 14279, Last known committed idx: 14278, Time since last communication: 0.000s
I20240503 08:12:06.864095 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 89. Current term is 90. Ops: []
I20240503 08:12:06.864409 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 89.14278, Next index: 14279, Last known committed idx: 14278, Time since last communication: 0.000s
I20240503 08:12:06.870021 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 90 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 89. Current term is 90. Ops: []
I20240503 08:12:06.870275 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 89.14278, Next index: 14279, Last known committed idx: 14278, Time since last communication: 0.000s
I20240503 08:12:06.870535 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.870584 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 90 FOLLOWER]: Advancing to term 91
I20240503 08:12:06.871268 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.871423 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 91 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.871588 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 91 candidate_status { last_received { term: 89 index: 14278 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.871598 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 91 candidate_status { last_received { term: 89 index: 14278 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.871654 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 90 FOLLOWER]: Advancing to term 91
I20240503 08:12:06.872243 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 91 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 91.
I20240503 08:12:06.872437 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 91 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:06.872542 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 91 FOLLOWER]: Leader election won for term 91
I20240503 08:12:06.872596 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 91 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:06.872644 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14278, Committed index: 14278, Last appended: 89.14278, Last appended by leader: 14278, Current term: 91, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.873142 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 89 to 91, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 91 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:06.882763 26516 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 89 LEADER]: Stepping down as leader of term 89
I20240503 08:12:06.882855 26516 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 89 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:06.882941 26516 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14278, Committed index: 14278, Last appended: 89.14278, Last appended by leader: 14278, Current term: 89, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.883049 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 89 FOLLOWER]: Advancing to term 90
W20240503 08:12:06.886704 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 87 current term: 91 is higher.
I20240503 08:12:06.898680 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 90 FOLLOWER]: Advancing to term 91
I20240503 08:12:06.899703 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 91 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 89 index: 14278. Preceding OpId from leader: term: 91 index: 14279. (index mismatch)
I20240503 08:12:06.900024 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14279, Last known committed idx: 14278, Time since last communication: 0.000s
I20240503 08:12:06.902798 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 91 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 89 index: 14278. Preceding OpId from leader: term: 91 index: 14279. (index mismatch)
I20240503 08:12:06.903023 26473 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14279, Last known committed idx: 14278, Time since last communication: 0.000s
W20240503 08:12:06.909900 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 87 current term: 91 is higher.
I20240503 08:12:06.964702 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 91 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:06.964812 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 91 FOLLOWER]: Advancing to term 92
I20240503 08:12:06.965544 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.965713 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 92 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:06.965847 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 92 candidate_status { last_received { term: 91 index: 14279 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:06.965896 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 92 candidate_status { last_received { term: 91 index: 14279 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.966130 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.966231 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I20240503 08:12:06.967249 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 92 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 91. Current term is 92. Ops: []
I20240503 08:12:06.967473 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 91.14279, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:06.986761 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 91 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:06.986862 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 91 FOLLOWER]: Advancing to term 92
I20240503 08:12:06.987635 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:06.987814 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 92 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:06.988015 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 92 candidate_status { last_received { term: 91 index: 14279 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:06.988062 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 92 candidate_status { last_received { term: 91 index: 14279 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:06.988134 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 92 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 92: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:06.988299 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 92 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:06.988399 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 92 FOLLOWER]: Leader election lost for term 92. Reason: could not achieve majority
I20240503 08:12:06.991976 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 92 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 91. Current term is 92. Ops: []
I20240503 08:12:06.992218 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 91.14279, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:06.999836 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 92 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:06.999955 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 92 FOLLOWER]: Advancing to term 93
I20240503 08:12:07.000648 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.000774 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 91. Current term is 93. Ops: []
I20240503 08:12:07.000809 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 93 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.000979 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 93 candidate_status { last_received { term: 91 index: 14279 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.001075 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 92 FOLLOWER]: Advancing to term 93
I20240503 08:12:07.001005 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 93 candidate_status { last_received { term: 91 index: 14279 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.000991 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 91.14279, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:07.001696 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 93.
I20240503 08:12:07.001992 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.002140 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 FOLLOWER]: Leader election won for term 93
I20240503 08:12:07.002219 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.002291 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14279, Committed index: 14279, Last appended: 91.14279, Last appended by leader: 14279, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.002801 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 91 to 93, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 93 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:07.022876 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 91. Current term is 93. Ops: []
I20240503 08:12:07.022876 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 93 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 91. Current term is 93. Ops: []
I20240503 08:12:07.023234 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 91.14279, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:07.023363 26517 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 93.14280 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14279
I20240503 08:12:07.023425 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 91.14279, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:07.029377 26284 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 91 LEADER]: Stepping down as leader of term 91
I20240503 08:12:07.029454 26284 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 91 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:07.029511 26284 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14279, Committed index: 14279, Last appended: 91.14279, Last appended by leader: 14279, Current term: 91, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.029618 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 91 FOLLOWER]: Advancing to term 93
I20240503 08:12:07.030256 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 93 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 91 index: 14279. Preceding OpId from leader: term: 93 index: 14280. (index mismatch)
I20240503 08:12:07.030511 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:07.036991 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 93 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 91 index: 14279. Preceding OpId from leader: term: 93 index: 14280. (index mismatch)
I20240503 08:12:07.037292 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14280, Last known committed idx: 14279, Time since last communication: 0.000s
I20240503 08:12:07.062108 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 93 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:07.062216 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 93 FOLLOWER]: Advancing to term 94
I20240503 08:12:07.062958 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.063156 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 94 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.063375 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 94 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.063388 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 94 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.063675 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.063763 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20240503 08:12:07.064666 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 94 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 93. Current term is 94. Ops: []
I20240503 08:12:07.064842 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.067706 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 93 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:07.067766 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 93 FOLLOWER]: Advancing to term 94
I20240503 08:12:07.068466 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.068603 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 94 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 93. Current term is 94. Ops: []
I20240503 08:12:07.068651 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 94 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.068794 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 94 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.068873 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 94: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:07.068794 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 94 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.068830 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.069044 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.069136 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20240503 08:12:07.098855 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 94 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 93. Current term is 94. Ops: []
I20240503 08:12:07.098937 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 LEADER]: Stepping down as leader of term 93
I20240503 08:12:07.099006 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.099110 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14280, Committed index: 14280, Last appended: 93.14280, Last appended by leader: 14280, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.099222 26384 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.099375 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 93 FOLLOWER]: Advancing to term 94
I20240503 08:12:07.101401 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 94 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.101466 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 94 FOLLOWER]: Advancing to term 95
I20240503 08:12:07.102031 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.102166 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 95 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.102350 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 95 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.102416 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 94 FOLLOWER]: Advancing to term 95
I20240503 08:12:07.102353 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 95 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.102450 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 94 FOLLOWER]: Advancing to term 95
I20240503 08:12:07.103044 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 95.
I20240503 08:12:07.103060 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 95.
I20240503 08:12:07.103237 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:07.103331 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 95 FOLLOWER]: Leader election won for term 95
I20240503 08:12:07.103390 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 95 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:07.103447 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14280, Committed index: 14280, Last appended: 93.14280, Last appended by leader: 14280, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.103921 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 93 to 95, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 95 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:07.122133 26473 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 93 current term: 95 is higher.
I20240503 08:12:07.129598 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 95 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 93 index: 14280. Preceding OpId from leader: term: 95 index: 14281. (index mismatch)
I20240503 08:12:07.129909 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.134475 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 95 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.134569 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 95 FOLLOWER]: Advancing to term 96
I20240503 08:12:07.135303 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.135473 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 96 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.135480 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 95. Current term is 96. Ops: []
I20240503 08:12:07.135650 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 96 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.135653 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 96 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.135864 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.135866 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.135998 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
W20240503 08:12:07.153542 26473 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 93 current term: 95 is higher.
I20240503 08:12:07.161582 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 95 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:07.161693 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 95 FOLLOWER]: Advancing to term 96
I20240503 08:12:07.162402 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.162578 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 96 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.162824 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 96 candidate_status { last_received { term: 95 index: 14281 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.162932 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 96: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:07.162829 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 96 candidate_status { last_received { term: 95 index: 14281 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.163123 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.163228 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
W20240503 08:12:07.163739 26473 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 93 current term: 95 is higher.
I20240503 08:12:07.166540 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 95. Current term is 96. Ops: [95.14281-95.14281]
I20240503 08:12:07.166759 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 96 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 95. Current term is 96. Ops: []
I20240503 08:12:07.166782 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.166987 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 95.14281, Next index: 14282, Last known committed idx: 14281, Time since last communication: 0.000s
I20240503 08:12:07.172950 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.173018 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 96 FOLLOWER]: Advancing to term 97
I20240503 08:12:07.173673 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.173821 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 97 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.174052 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 97 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.174058 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 97 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.174141 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 96 FOLLOWER]: Advancing to term 97
I20240503 08:12:07.174799 26133 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 97 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 97 because replica has last-logged OpId of term: 95 index: 14281, which is greater than that of the candidate, which has last-logged OpId of term: 93 index: 14280.
I20240503 08:12:07.175009 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 97 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.175123 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 97 FOLLOWER]: Leader election lost for term 97. Reason: could not achieve majority
I20240503 08:12:07.189123 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 97 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 95. Current term is 97. Ops: [95.14281-95.14281]
I20240503 08:12:07.189399 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.190143 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 97 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 95. Current term is 97. Ops: [95.14281-95.14281]
I20240503 08:12:07.190363 26473 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.202836 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 97 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.202953 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 97 FOLLOWER]: Advancing to term 98
I20240503 08:12:07.203747 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.203897 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 95. Current term is 98. Ops: []
I20240503 08:12:07.203939 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 98 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.204088 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 98 candidate_status { last_received { term: 95 index: 14281 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.204089 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 98 candidate_status { last_received { term: 95 index: 14281 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.204216 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 97 FOLLOWER]: Advancing to term 98
I20240503 08:12:07.204142 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 95.14281, Next index: 14282, Last known committed idx: 14281, Time since last communication: 0.000s
I20240503 08:12:07.204824 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 98.
I20240503 08:12:07.205029 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.205119 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 FOLLOWER]: Leader election won for term 98
I20240503 08:12:07.205186 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:07.205240 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14281, Committed index: 14281, Last appended: 95.14281, Last appended by leader: 14281, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.205904 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 95 to 98, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 98 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:07.210244 26481 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 95 LEADER]: Stepping down as leader of term 95
I20240503 08:12:07.210312 26481 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 95 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:07.210368 26481 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14281, Committed index: 14281, Last appended: 95.14281, Last appended by leader: 14281, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.210489 26481 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 95 FOLLOWER]: Advancing to term 96
I20240503 08:12:07.234628 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 96 FOLLOWER]: Advancing to term 98
I20240503 08:12:07.235426 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 98 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 95 index: 14281. Preceding OpId from leader: term: 98 index: 14282. (index mismatch)
I20240503 08:12:07.235867 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14282, Last known committed idx: 14281, Time since last communication: 0.000s
I20240503 08:12:07.236483 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 98 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.236686 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 98 FOLLOWER]: Advancing to term 99
I20240503 08:12:07.237581 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.237730 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 99 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.237905 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 99 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.237912 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 99 candidate_status { last_received { term: 93 index: 14280 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.238234 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.238355 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20240503 08:12:07.240878 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 98. Current term is 99. Ops: []
I20240503 08:12:07.241225 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14282, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.256548 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 98. Current term is 99. Ops: [95.14281-98.14282]
I20240503 08:12:07.256836 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.267834 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 98 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:07.267969 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 98 FOLLOWER]: Advancing to term 99
I20240503 08:12:07.268674 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.268910 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 99 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.268936 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 99 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 98. Current term is 99. Ops: []
I20240503 08:12:07.268950 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 99 candidate_status { last_received { term: 98 index: 14282 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.269042 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 99 candidate_status { last_received { term: 98 index: 14282 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.269116 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 99: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:07.269196 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 98.14282, Next index: 14283, Last known committed idx: 14282, Time since last communication: 0.000s
I20240503 08:12:07.269253 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.269333 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20240503 08:12:07.284788 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 98. Current term is 99. Ops: [95.14281-98.14282]
I20240503 08:12:07.285094 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.294373 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 99 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 98. Current term is 99. Ops: []
I20240503 08:12:07.294672 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 98.14282, Next index: 14283, Last known committed idx: 14282, Time since last communication: 0.000s
I20240503 08:12:07.311090 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.311190 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 99 FOLLOWER]: Advancing to term 100
I20240503 08:12:07.311868 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.312039 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 100 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.312259 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 100 candidate_status { last_received { term: 98 index: 14282 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.312275 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 100 candidate_status { last_received { term: 98 index: 14282 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.312359 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 99 FOLLOWER]: Advancing to term 100
I20240503 08:12:07.312973 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 100.
I20240503 08:12:07.313164 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.313266 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 100 FOLLOWER]: Leader election won for term 100
I20240503 08:12:07.313325 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 100 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:07.313369 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14282, Committed index: 14282, Last appended: 98.14282, Last appended by leader: 14282, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.313800 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 98 to 100, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 100 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:07.314033 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 100 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 98. Current term is 100. Ops: [95.14281-98.14282]
I20240503 08:12:07.314359 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 93.14280, Next index: 14281, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.317420 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 LEADER]: Stepping down as leader of term 98
I20240503 08:12:07.317476 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:07.317528 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14282, Committed index: 14282, Last appended: 98.14282, Last appended by leader: 14282, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.317623 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 98 FOLLOWER]: Advancing to term 99
I20240503 08:12:07.340021 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.340127 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 99 FOLLOWER]: Advancing to term 100
W20240503 08:12:07.340806 26481 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 98 current term: 100 is higher.
I20240503 08:12:07.340848 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.341018 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 100 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.341264 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 100 candidate_status { last_received { term: 98 index: 14282 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.341311 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 100 candidate_status { last_received { term: 98 index: 14282 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.341392 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 100 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 100: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:07.341547 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 100 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.341650 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 100 FOLLOWER]: Leader election lost for term 100. Reason: could not achieve majority
I20240503 08:12:07.342305 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 100 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 98 index: 14282. Preceding OpId from leader: term: 100 index: 14283. (index mismatch)
I20240503 08:12:07.342540 26481 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14283, Last known committed idx: 14282, Time since last communication: 0.000s
I20240503 08:12:07.348088 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 100 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 93 index: 14280. Preceding OpId from leader: term: 100 index: 14283. (index mismatch)
I20240503 08:12:07.348469 26481 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14283, Last known committed idx: 14280, Time since last communication: 0.000s
I20240503 08:12:07.374368 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 100 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:07.374465 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 100 FOLLOWER]: Advancing to term 101
I20240503 08:12:07.375160 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.375326 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 101 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.375520 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 101 candidate_status { last_received { term: 100 index: 14283 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.375526 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 101 candidate_status { last_received { term: 100 index: 14283 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.375790 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.375878 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20240503 08:12:07.377995 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 101 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 100. Current term is 101. Ops: []
I20240503 08:12:07.378187 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 100.14283, Next index: 14284, Last known committed idx: 14283, Time since last communication: 0.000s
I20240503 08:12:07.406548 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 101 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.406698 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 101 FOLLOWER]: Advancing to term 102
I20240503 08:12:07.407495 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.407670 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 102 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.407889 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 102 candidate_status { last_received { term: 100 index: 14283 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.407887 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 102 candidate_status { last_received { term: 100 index: 14283 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.407987 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 100 FOLLOWER]: Advancing to term 102
I20240503 08:12:07.408617 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 102.
I20240503 08:12:07.408857 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.408888 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 100. Current term is 102. Ops: []
I20240503 08:12:07.408967 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 FOLLOWER]: Leader election won for term 102
I20240503 08:12:07.409029 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:07.409076 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14283, Committed index: 14283, Last appended: 100.14283, Last appended by leader: 14283, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.409379 26481 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 100.14283, Next index: 14284, Last known committed idx: 14283, Time since last communication: 0.000s
I20240503 08:12:07.409464 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 100 to 102, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 102 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:07.409698 26502 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 99 current term: 102 is higher.
I20240503 08:12:07.414409 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 100 LEADER]: Stepping down as leader of term 100
I20240503 08:12:07.414508 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 100 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:07.414620 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 102 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 100. Current term is 102. Ops: []
I20240503 08:12:07.414568 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14283, Committed index: 14283, Last appended: 100.14283, Last appended by leader: 14283, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.414718 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 100 FOLLOWER]: Advancing to term 101
I20240503 08:12:07.432014 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 102 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 100 index: 14283. Preceding OpId from leader: term: 102 index: 14284. (index mismatch)
I20240503 08:12:07.432292 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14284, Last known committed idx: 14283, Time since last communication: 0.000s
I20240503 08:12:07.432911 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 101 FOLLOWER]: Advancing to term 102
I20240503 08:12:07.433769 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 102 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 100 index: 14283. Preceding OpId from leader: term: 102 index: 14284. (index mismatch)
I20240503 08:12:07.434054 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14284, Last known committed idx: 14283, Time since last communication: 0.000s
W20240503 08:12:07.471994 26502 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 99 current term: 102 is higher.
I20240503 08:12:07.488168 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 102 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:07.488269 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 102 FOLLOWER]: Advancing to term 103
I20240503 08:12:07.488945 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.489100 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 103 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.489334 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 103 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.489356 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 103 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.489581 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.489701 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20240503 08:12:07.494089 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 103 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 102. Current term is 103. Ops: []
I20240503 08:12:07.494405 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 102.14284, Next index: 14285, Last known committed idx: 14284, Time since last communication: 0.000s
W20240503 08:12:07.511219 26502 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 100 current term: 102 is higher.
I20240503 08:12:07.522339 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.522434 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 103 FOLLOWER]: Advancing to term 104
I20240503 08:12:07.523128 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.523283 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 104 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 102. Current term is 104. Ops: []
I20240503 08:12:07.523371 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 104 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.523440 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 104 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.523496 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 104 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.523625 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 102.14284, Next index: 14285, Last known committed idx: 14284, Time since last communication: 0.000s
I20240503 08:12:07.523674 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 104 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.523769 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 104 FOLLOWER]: Leader election lost for term 104. Reason: could not achieve majority
I20240503 08:12:07.555472 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 104 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 102. Current term is 104. Ops: []
I20240503 08:12:07.555716 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 102.14284, Next index: 14285, Last known committed idx: 14284, Time since last communication: 0.000s
I20240503 08:12:07.565263 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 104 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.565352 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 104 FOLLOWER]: Advancing to term 105
I20240503 08:12:07.566032 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.566238 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 105 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.566437 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 105 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.566445 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 105 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.566799 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.566920 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20240503 08:12:07.585036 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 105 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 102. Current term is 105. Ops: []
I20240503 08:12:07.585322 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 102.14284, Next index: 14285, Last known committed idx: 14284, Time since last communication: 0.000s
I20240503 08:12:07.605672 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.605790 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 105 FOLLOWER]: Advancing to term 106
I20240503 08:12:07.606472 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.606647 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 106 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.606907 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 106 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.606906 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 106 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.607245 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 106 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.607376 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election lost for term 106. Reason: could not achieve majority
I20240503 08:12:07.607928 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 LEADER]: Stepping down as leader of term 102
I20240503 08:12:07.607971 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:07.608036 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14284, Committed index: 14284, Last appended: 102.14284, Last appended by leader: 14284, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.608127 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 102 FOLLOWER]: Advancing to term 103
I20240503 08:12:07.610466 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 102 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:07.610522 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 102 FOLLOWER]: Advancing to term 103
I20240503 08:12:07.611088 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.611263 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 103 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.611400 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 103 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.611474 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 103 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.611522 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 103. Current term is 106.
I20240503 08:12:07.611685 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 103 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 103. Current term is 106.
I20240503 08:12:07.611732 26173 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 103 election: Cancelling election due to peer responding with higher term
I20240503 08:12:07.611783 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 103 FOLLOWER]: Advancing to term 106
I20240503 08:12:07.612056 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 103.
I20240503 08:12:07.612265 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 106 FOLLOWER]: Leader election lost for term 103. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 103. Current term is 106.
I20240503 08:12:07.642728 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.642845 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 103 FOLLOWER]: Advancing to term 104
I20240503 08:12:07.643543 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.643752 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 104 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.643914 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 104 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.644001 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 104. Current term is 106.
I20240503 08:12:07.644042 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 104 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.644099 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 104. Current term is 106.
I20240503 08:12:07.644173 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 104 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 104. Current term is 106.
I20240503 08:12:07.644214 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 104 election: Cancelling election due to peer responding with higher term
I20240503 08:12:07.644258 26098 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 104 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 104. Current term is 106.
I20240503 08:12:07.644397 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 104 FOLLOWER]: Advancing to term 106
I20240503 08:12:07.645040 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 106 FOLLOWER]: Leader election lost for term 104. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 104. Current term is 106.
I20240503 08:12:07.646385 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 106 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.646432 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 106 FOLLOWER]: Advancing to term 107
I20240503 08:12:07.647039 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.647179 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 107 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.647359 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 107 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.647469 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 106 FOLLOWER]: Advancing to term 107
I20240503 08:12:07.647359 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 107 candidate_status { last_received { term: 102 index: 14284 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.647517 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 106 FOLLOWER]: Advancing to term 107
I20240503 08:12:07.648135 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 107.
I20240503 08:12:07.648135 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 107.
I20240503 08:12:07.648340 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:07.648429 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 107 FOLLOWER]: Leader election won for term 107
I20240503 08:12:07.648514 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 107 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.648569 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14284, Committed index: 14284, Last appended: 102.14284, Last appended by leader: 14284, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.649065 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 102 to 107, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 107 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:07.678575 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 107 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 102 index: 14284. Preceding OpId from leader: term: 107 index: 14285. (index mismatch)
I20240503 08:12:07.678875 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14285, Last known committed idx: 14284, Time since last communication: 0.000s
I20240503 08:12:07.680380 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 107 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 102 index: 14284. Preceding OpId from leader: term: 107 index: 14285. (index mismatch)
I20240503 08:12:07.680608 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14285, Last known committed idx: 14284, Time since last communication: 0.000s
I20240503 08:12:07.711010 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 107 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:07.711109 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 107 FOLLOWER]: Advancing to term 108
I20240503 08:12:07.711843 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.712007 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 107. Current term is 108. Ops: []
I20240503 08:12:07.712021 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 108 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.712158 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 108 candidate_status { last_received { term: 107 index: 14285 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.712133 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 108 candidate_status { last_received { term: 107 index: 14285 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.712385 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 108 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.712364 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 107.14285, Next index: 14286, Last known committed idx: 14285, Time since last communication: 0.000s
I20240503 08:12:07.712491 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Leader election lost for term 108. Reason: could not achieve majority
I20240503 08:12:07.738106 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 107. Current term is 108. Ops: []
I20240503 08:12:07.738381 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 107.14285, Next index: 14286, Last known committed idx: 14285, Time since last communication: 0.000s
I20240503 08:12:07.739179 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 107 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:07.739253 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 107 FOLLOWER]: Advancing to term 108
I20240503 08:12:07.739974 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.740130 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 108 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.740352 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 108 candidate_status { last_received { term: 107 index: 14285 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.740357 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 108 candidate_status { last_received { term: 107 index: 14285 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.740470 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 108: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:07.740721 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 108 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.740867 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 108 FOLLOWER]: Leader election lost for term 108. Reason: could not achieve majority
I20240503 08:12:07.742399 26531 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 107 LEADER]: Stepping down as leader of term 107
I20240503 08:12:07.742451 26531 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 107 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.742532 26531 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14285, Committed index: 14285, Last appended: 107.14285, Last appended by leader: 14285, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.742672 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 107 FOLLOWER]: Advancing to term 108
I20240503 08:12:07.744036 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.744120 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 108 FOLLOWER]: Advancing to term 109
I20240503 08:12:07.744752 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.744906 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 109 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.745139 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 109 candidate_status { last_received { term: 107 index: 14285 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.745139 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 109 candidate_status { last_received { term: 107 index: 14285 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.745271 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 108 FOLLOWER]: Advancing to term 109
I20240503 08:12:07.745275 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 108 FOLLOWER]: Advancing to term 109
I20240503 08:12:07.745963 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 109.
I20240503 08:12:07.745972 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 109.
I20240503 08:12:07.746330 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:07.746456 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 109 FOLLOWER]: Leader election won for term 109
I20240503 08:12:07.746519 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 109 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:07.746584 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14285, Committed index: 14285, Last appended: 107.14285, Last appended by leader: 14285, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.747104 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 107 to 109, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 109 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:07.775673 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 109 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 107 index: 14285. Preceding OpId from leader: term: 109 index: 14286. (index mismatch)
I20240503 08:12:07.775867 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 109 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 107 index: 14285. Preceding OpId from leader: term: 109 index: 14286. (index mismatch)
I20240503 08:12:07.775995 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14286, Last known committed idx: 14285, Time since last communication: 0.000s
I20240503 08:12:07.776132 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14286, Last known committed idx: 14285, Time since last communication: 0.000s
W20240503 08:12:07.805233 26349 lock_manager.cc:494] Txn 5 has not acquired the partition lock after 1000ms
I20240503 08:12:07.807291 26531 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 109 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:07.807360 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 109 FOLLOWER]: Advancing to term 110
I20240503 08:12:07.808079 26531 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.808220 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 109. Current term is 110. Ops: []
I20240503 08:12:07.808257 26531 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 110 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.808269 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 109 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:07.808351 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 109 FOLLOWER]: Advancing to term 110
I20240503 08:12:07.808459 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 110 candidate_status { last_received { term: 109 index: 14286 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.808470 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 110 candidate_status { last_received { term: 109 index: 14286 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.808478 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 109.14286, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.809034 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.809125 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 110: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:07.809155 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 110 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.809290 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 110 candidate_status { last_received { term: 109 index: 14286 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.809296 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 110 candidate_status { last_received { term: 109 index: 14286 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.809296 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 110 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.809408 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 110: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:07.809546 26531 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Leader election lost for term 110. Reason: could not achieve majority
I20240503 08:12:07.809561 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 110 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.809634 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 110 FOLLOWER]: Leader election lost for term 110. Reason: could not achieve majority
I20240503 08:12:07.812916 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 110 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 109. Current term is 110. Ops: []
I20240503 08:12:07.813190 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 109.14286, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.833382 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 109. Current term is 110. Ops: []
I20240503 08:12:07.833688 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 109.14286, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.844172 26531 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.844281 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 110 FOLLOWER]: Advancing to term 111
I20240503 08:12:07.845140 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 110 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 109. Current term is 110. Ops: []
I20240503 08:12:07.845146 26531 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.845372 26531 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 111 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.845461 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 109.14286, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.845466 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 111 candidate_status { last_received { term: 109 index: 14286 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.845521 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 111 candidate_status { last_received { term: 109 index: 14286 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.845664 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 110 FOLLOWER]: Advancing to term 111
I20240503 08:12:07.846210 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 111.
I20240503 08:12:07.846406 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:07.846544 26531 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 FOLLOWER]: Leader election won for term 111
I20240503 08:12:07.846601 26531 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.846698 26531 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14286, Committed index: 14286, Last appended: 109.14286, Last appended by leader: 14286, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.847226 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 109 to 111, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 111 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:07.856212 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 109. Current term is 111. Ops: []
I20240503 08:12:07.856492 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 109.14286, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.856598 26502 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 111.14287 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14286
I20240503 08:12:07.870718 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 109 LEADER]: Stepping down as leader of term 109
I20240503 08:12:07.870781 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 109 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:07.870826 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14286, Committed index: 14286, Last appended: 109.14286, Last appended by leader: 14286, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.870925 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 109 FOLLOWER]: Advancing to term 111
I20240503 08:12:07.871577 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 111 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 109 index: 14286. Preceding OpId from leader: term: 111 index: 14287. (index mismatch)
I20240503 08:12:07.871829 26531 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.877175 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 111 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 109 index: 14286. Preceding OpId from leader: term: 111 index: 14287. (index mismatch)
I20240503 08:12:07.877421 26531 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14287, Last known committed idx: 14286, Time since last communication: 0.000s
I20240503 08:12:07.931226 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 111 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:07.931330 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 111 FOLLOWER]: Advancing to term 112
I20240503 08:12:07.932057 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.932238 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 112 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.932415 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 112 candidate_status { last_received { term: 111 index: 14287 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.932441 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 112 candidate_status { last_received { term: 111 index: 14287 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.932677 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:07.932773 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20240503 08:12:07.932904 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 111 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:07.932966 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 111 FOLLOWER]: Advancing to term 112
I20240503 08:12:07.933543 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.933732 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 112 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:07.933809 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 112 candidate_status { last_received { term: 111 index: 14287 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:07.933809 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 112 candidate_status { last_received { term: 111 index: 14287 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.933977 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 112: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:07.934069 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 112 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 111. Current term is 112. Ops: []
I20240503 08:12:07.934245 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:07.934324 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 111.14287, Next index: 14288, Last known committed idx: 14287, Time since last communication: 0.000s
I20240503 08:12:07.934372 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20240503 08:12:07.935497 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 LEADER]: Stepping down as leader of term 111
I20240503 08:12:07.935541 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.935604 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14287, Committed index: 14287, Last appended: 111.14287, Last appended by leader: 14287, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.935678 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 111 FOLLOWER]: Advancing to term 112
I20240503 08:12:07.969283 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 112 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:07.969383 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 112 FOLLOWER]: Advancing to term 113
I20240503 08:12:07.970052 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.970217 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 113 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:07.970445 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 113 candidate_status { last_received { term: 111 index: 14287 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:07.970445 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 113 candidate_status { last_received { term: 111 index: 14287 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:07.970568 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 112 FOLLOWER]: Advancing to term 113
I20240503 08:12:07.970584 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 112 FOLLOWER]: Advancing to term 113
I20240503 08:12:07.971262 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 113.
I20240503 08:12:07.971262 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 113.
I20240503 08:12:07.971483 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:07.971588 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 113 FOLLOWER]: Leader election won for term 113
I20240503 08:12:07.971650 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 113 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:07.971684 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14287, Committed index: 14287, Last appended: 111.14287, Last appended by leader: 14287, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:07.972179 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 111 to 113. New cstate: current_term: 113 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:07.994458 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 113 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 111 index: 14287. Preceding OpId from leader: term: 113 index: 14288. (index mismatch)
I20240503 08:12:07.994761 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14288, Last known committed idx: 14287, Time since last communication: 0.000s
I20240503 08:12:07.995803 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 113 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 111 index: 14287. Preceding OpId from leader: term: 113 index: 14288. (index mismatch)
I20240503 08:12:07.996022 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14288, Last known committed idx: 14287, Time since last communication: 0.000s
I20240503 08:12:08.054314 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 113 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.054421 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 113 FOLLOWER]: Advancing to term 114
I20240503 08:12:08.055156 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.055337 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 114 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.055528 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 114 candidate_status { last_received { term: 113 index: 14288 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.055553 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 114 candidate_status { last_received { term: 113 index: 14288 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.055763 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.055859 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20240503 08:12:08.057085 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 114 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 113. Current term is 114. Ops: []
I20240503 08:12:08.057302 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 113.14288, Next index: 14289, Last known committed idx: 14288, Time since last communication: 0.000s
I20240503 08:12:08.057480 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 113 LEADER]: Stepping down as leader of term 113
I20240503 08:12:08.057509 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 113 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.057555 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14288, Committed index: 14288, Last appended: 113.14288, Last appended by leader: 14288, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.057638 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 113 FOLLOWER]: Advancing to term 114
I20240503 08:12:08.078979 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 113 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.079092 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 113 FOLLOWER]: Advancing to term 114
I20240503 08:12:08.079813 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.079993 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 114 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.080077 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 114 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.080113 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 114 FOLLOWER]: Advancing to term 115
I20240503 08:12:08.080207 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 114 candidate_status { last_received { term: 113 index: 14288 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.080196 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 114 candidate_status { last_received { term: 113 index: 14288 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.080381 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 114: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:08.080859 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.081017 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 114. Current term is 115.
I20240503 08:12:08.081118 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 115 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.081159 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 115 candidate_status { last_received { term: 113 index: 14288 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.081225 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 114 FOLLOWER]: Advancing to term 115
I20240503 08:12:08.081241 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 114 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 114. Current term is 115.
I20240503 08:12:08.081267 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 114 election: Cancelling election due to peer responding with higher term
I20240503 08:12:08.081274 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 115 candidate_status { last_received { term: 113 index: 14288 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.081353 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 114 FOLLOWER]: Advancing to term 115
I20240503 08:12:08.081821 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 115.
I20240503 08:12:08.081881 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 115.
I20240503 08:12:08.081959 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 115 FOLLOWER]: Leader election lost for term 114. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 114. Current term is 115.
I20240503 08:12:08.082017 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:08.082111 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 FOLLOWER]: Leader election won for term 115
I20240503 08:12:08.082151 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.082217 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14288, Committed index: 14288, Last appended: 113.14288, Last appended by leader: 14288, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.082859 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 113 to 115. New cstate: current_term: 115 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:08.110289 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 115 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 113 index: 14288. Preceding OpId from leader: term: 115 index: 14289. (index mismatch)
I20240503 08:12:08.110644 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14289, Last known committed idx: 14288, Time since last communication: 0.000s
I20240503 08:12:08.118402 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 115 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 113 index: 14288. Preceding OpId from leader: term: 115 index: 14289. (index mismatch)
I20240503 08:12:08.118769 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14289, Last known committed idx: 14288, Time since last communication: 0.000s
I20240503 08:12:08.142577 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 115 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.142701 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 115 FOLLOWER]: Advancing to term 116
I20240503 08:12:08.143407 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.143594 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 116 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.143811 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 116 candidate_status { last_received { term: 115 index: 14289 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.143911 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 116 candidate_status { last_received { term: 115 index: 14289 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.144232 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.144392 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20240503 08:12:08.147626 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 115. Current term is 116. Ops: []
I20240503 08:12:08.147934 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 115.14289, Next index: 14290, Last known committed idx: 14289, Time since last communication: 0.000s
I20240503 08:12:08.149170 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 115 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.149235 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 115 FOLLOWER]: Advancing to term 116
I20240503 08:12:08.149972 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.150111 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 116 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.150333 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 116 candidate_status { last_received { term: 115 index: 14289 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.150370 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 116 candidate_status { last_received { term: 115 index: 14289 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.150431 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 116: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:08.150650 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:08.150779 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20240503 08:12:08.151965 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 116 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 115. Current term is 116. Ops: []
I20240503 08:12:08.152212 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 115.14289, Next index: 14290, Last known committed idx: 14289, Time since last communication: 0.000s
I20240503 08:12:08.171519 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 115. Current term is 116. Ops: []
I20240503 08:12:08.171833 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 115.14289, Next index: 14290, Last known committed idx: 14289, Time since last communication: 0.000s
I20240503 08:12:08.174590 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 116 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 115. Current term is 116. Ops: []
I20240503 08:12:08.174799 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 115.14289, Next index: 14290, Last known committed idx: 14289, Time since last communication: 0.000s
I20240503 08:12:08.177023 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.177078 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 116 FOLLOWER]: Advancing to term 117
I20240503 08:12:08.177738 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.177865 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 117 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.178052 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 117 candidate_status { last_received { term: 115 index: 14289 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.178117 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 116 FOLLOWER]: Advancing to term 117
I20240503 08:12:08.178056 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 117 candidate_status { last_received { term: 115 index: 14289 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.178714 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 117.
I20240503 08:12:08.178939 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:08.179039 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 117 FOLLOWER]: Leader election won for term 117
I20240503 08:12:08.179087 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 117 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:08.179147 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14289, Committed index: 14289, Last appended: 115.14289, Last appended by leader: 14289, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.179567 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 115 to 117, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 117 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:08.182165 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 LEADER]: Stepping down as leader of term 115
I20240503 08:12:08.182209 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.182267 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14289, Committed index: 14289, Last appended: 115.14289, Last appended by leader: 14289, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.182351 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 115 FOLLOWER]: Advancing to term 116
I20240503 08:12:08.201819 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 117 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 115 index: 14289. Preceding OpId from leader: term: 117 index: 14290. (index mismatch)
I20240503 08:12:08.202200 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14290, Last known committed idx: 14289, Time since last communication: 0.000s
I20240503 08:12:08.204761 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 116 FOLLOWER]: Advancing to term 117
I20240503 08:12:08.205451 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 117 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 115 index: 14289. Preceding OpId from leader: term: 117 index: 14290. (index mismatch)
I20240503 08:12:08.205704 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14290, Last known committed idx: 14289, Time since last communication: 0.000s
I20240503 08:12:08.233866 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 117 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:08.233963 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 117 FOLLOWER]: Advancing to term 118
I20240503 08:12:08.234709 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.234894 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 118 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.235142 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 118 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.235188 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 118 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.235461 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:08.235630 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20240503 08:12:08.236701 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 117 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:08.236941 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 117 FOLLOWER]: Advancing to term 118
I20240503 08:12:08.237646 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.237884 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 118 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.238114 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 118 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.238114 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 118 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.238226 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 118: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:08.238505 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.238633 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20240503 08:12:08.239485 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 118. Ops: []
I20240503 08:12:08.239724 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.241362 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 118 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 118. Ops: []
I20240503 08:12:08.241681 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.265668 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 118 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 118. Ops: []
I20240503 08:12:08.265990 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.270474 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 118. Ops: []
I20240503 08:12:08.270756 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.276429 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.276525 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 118 FOLLOWER]: Advancing to term 119
I20240503 08:12:08.276834 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.276909 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 118 FOLLOWER]: Advancing to term 119
I20240503 08:12:08.277232 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.277420 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 119 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.277539 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.277593 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 119 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.277602 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 119 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.277680 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 119 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.277681 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 119 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 119: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:08.277799 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 119 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.277837 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 119 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.277904 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 119 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 119: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:08.277992 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 119 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.278067 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 119 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:08.278095 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 119 FOLLOWER]: Leader election lost for term 119. Reason: could not achieve majority
I20240503 08:12:08.278159 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 119 FOLLOWER]: Leader election lost for term 119. Reason: could not achieve majority
I20240503 08:12:08.291745 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 119 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 119. Ops: []
I20240503 08:12:08.292016 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.303614 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 119 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 119. Ops: []
I20240503 08:12:08.303900 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.315889 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 119 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 117. Current term is 119. Ops: []
I20240503 08:12:08.316195 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 117.14290, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.318555 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 117 LEADER]: Stepping down as leader of term 117
I20240503 08:12:08.318637 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 117 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:08.318714 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14290, Committed index: 14290, Last appended: 117.14290, Last appended by leader: 14290, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.318815 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 117 FOLLOWER]: Advancing to term 118
I20240503 08:12:08.330085 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 119 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.330199 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 119 FOLLOWER]: Advancing to term 120
I20240503 08:12:08.330936 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.331113 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 120 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.331372 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 120 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.331342 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 120 candidate_status { last_received { term: 117 index: 14290 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.331458 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 119 FOLLOWER]: Advancing to term 120
I20240503 08:12:08.331462 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 118 FOLLOWER]: Advancing to term 120
I20240503 08:12:08.332095 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 120.
I20240503 08:12:08.332096 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 120.
I20240503 08:12:08.332329 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:08.332451 26538 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 120 FOLLOWER]: Leader election won for term 120
I20240503 08:12:08.332518 26538 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 120 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:08.332561 26538 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14290, Committed index: 14290, Last appended: 117.14290, Last appended by leader: 14290, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.333020 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 117 to 120, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 120 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:08.361624 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 120 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 117 index: 14290. Preceding OpId from leader: term: 120 index: 14291. (index mismatch)
I20240503 08:12:08.361886 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.366449 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 120 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 117 index: 14290. Preceding OpId from leader: term: 120 index: 14291. (index mismatch)
I20240503 08:12:08.366696 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14291, Last known committed idx: 14290, Time since last communication: 0.000s
I20240503 08:12:08.393734 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 120 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:08.393831 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 120 FOLLOWER]: Advancing to term 121
I20240503 08:12:08.394492 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.394629 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 120. Current term is 121. Ops: []
I20240503 08:12:08.394687 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 121 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.394840 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 121 candidate_status { last_received { term: 120 index: 14291 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.394858 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 120.14291, Next index: 14292, Last known committed idx: 14291, Time since last communication: 0.000s
I20240503 08:12:08.394845 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 121 candidate_status { last_received { term: 120 index: 14291 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.395102 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 121 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.395195 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Leader election lost for term 121. Reason: could not achieve majority
I20240503 08:12:08.421734 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 120. Current term is 121. Ops: []
I20240503 08:12:08.422029 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 120.14291, Next index: 14292, Last known committed idx: 14291, Time since last communication: 0.000s
I20240503 08:12:08.423419 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 120 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:08.423481 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 120 FOLLOWER]: Advancing to term 121
I20240503 08:12:08.424089 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.424232 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 121 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.424441 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 121 candidate_status { last_received { term: 120 index: 14291 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.424465 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 121 candidate_status { last_received { term: 120 index: 14291 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.424540 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 121: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:08.424731 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 121 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.424841 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 121 FOLLOWER]: Leader election lost for term 121. Reason: could not achieve majority
I20240503 08:12:08.425498 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 121 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 120. Current term is 121. Ops: []
I20240503 08:12:08.425741 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 120.14291, Next index: 14292, Last known committed idx: 14291, Time since last communication: 0.000s
I20240503 08:12:08.434051 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.434176 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 121 FOLLOWER]: Advancing to term 122
I20240503 08:12:08.434999 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.435180 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 122 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.435446 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 122 candidate_status { last_received { term: 120 index: 14291 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.435559 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 121 FOLLOWER]: Advancing to term 122
I20240503 08:12:08.435451 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 122 candidate_status { last_received { term: 120 index: 14291 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.436326 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 122.
I20240503 08:12:08.436540 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.436647 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 122 FOLLOWER]: Leader election won for term 122
I20240503 08:12:08.436700 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 122 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.436764 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14291, Committed index: 14291, Last appended: 120.14291, Last appended by leader: 14291, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.437345 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 120 to 122, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 122 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:08.446337 26538 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 120 LEADER]: Stepping down as leader of term 120
I20240503 08:12:08.446451 26538 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 120 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:08.446532 26538 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14291, Committed index: 14291, Last appended: 120.14291, Last appended by leader: 14291, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.446691 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 120 FOLLOWER]: Advancing to term 121
I20240503 08:12:08.459545 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 121 FOLLOWER]: Advancing to term 122
I20240503 08:12:08.460258 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 122 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 120 index: 14291. Preceding OpId from leader: term: 122 index: 14292. (index mismatch)
I20240503 08:12:08.460556 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14292, Last known committed idx: 14291, Time since last communication: 0.000s
I20240503 08:12:08.469034 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 122 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 120 index: 14291. Preceding OpId from leader: term: 122 index: 14292. (index mismatch)
I20240503 08:12:08.469310 26531 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14292, Last known committed idx: 14291, Time since last communication: 0.000s
I20240503 08:12:08.492528 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 122 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.492642 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 122 FOLLOWER]: Advancing to term 123
I20240503 08:12:08.493311 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.493484 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 123 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.493705 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 123 candidate_status { last_received { term: 122 index: 14292 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.493727 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 123 candidate_status { last_received { term: 122 index: 14292 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.493953 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:08.494057 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20240503 08:12:08.497083 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 122. Current term is 123. Ops: []
I20240503 08:12:08.497334 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 122.14292, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.500682 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 122 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.500744 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 122 FOLLOWER]: Advancing to term 123
I20240503 08:12:08.501430 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.501571 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 123 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.501753 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 123 candidate_status { last_received { term: 122 index: 14292 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.501753 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 123 candidate_status { last_received { term: 122 index: 14292 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.501871 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 123: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:08.502043 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 123 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.502144 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 123 FOLLOWER]: Leader election lost for term 123. Reason: could not achieve majority
I20240503 08:12:08.503762 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 123 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 122. Current term is 123. Ops: []
I20240503 08:12:08.504014 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 122.14292, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.526453 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 122. Current term is 123. Ops: []
I20240503 08:12:08.526764 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 122.14292, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.532461 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.532560 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 123 FOLLOWER]: Advancing to term 124
I20240503 08:12:08.533633 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.533871 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 124 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.534071 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 124 candidate_status { last_received { term: 122 index: 14292 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.534286 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 124 candidate_status { last_received { term: 122 index: 14292 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.534375 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 123 FOLLOWER]: Advancing to term 124
I20240503 08:12:08.535044 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 124.
I20240503 08:12:08.535250 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:08.535379 26538 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 FOLLOWER]: Leader election won for term 124
I20240503 08:12:08.535429 26538 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:08.535477 26538 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14292, Committed index: 14292, Last appended: 122.14292, Last appended by leader: 14292, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.535935 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 122 to 124, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 124 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:08.536418 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 124 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 122. Current term is 124. Ops: []
I20240503 08:12:08.536623 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 122.14292, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.559965 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 122. Current term is 124. Ops: []
I20240503 08:12:08.560359 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 122.14292, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.560493 26531 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 124.14293 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14292
I20240503 08:12:08.562718 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 122 LEADER]: Stepping down as leader of term 122
I20240503 08:12:08.562769 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 122 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.562831 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14292, Committed index: 14292, Last appended: 122.14292, Last appended by leader: 14292, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.562932 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 122 FOLLOWER]: Advancing to term 123
I20240503 08:12:08.565395 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 124 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 122 index: 14292. Preceding OpId from leader: term: 124 index: 14293. (index mismatch)
I20240503 08:12:08.565634 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.570009 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 123 FOLLOWER]: Advancing to term 124
I20240503 08:12:08.570741 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 124 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 122 index: 14292. Preceding OpId from leader: term: 124 index: 14293. (index mismatch)
I20240503 08:12:08.570994 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14293, Last known committed idx: 14292, Time since last communication: 0.000s
I20240503 08:12:08.623143 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 124 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:08.623250 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 124 FOLLOWER]: Advancing to term 125
I20240503 08:12:08.623950 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.624110 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 125 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.624119 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 125 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 124. Current term is 125. Ops: []
I20240503 08:12:08.624284 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 125 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.624344 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 125 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.624528 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 125 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.624539 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 124.14293, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.624620 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 125 FOLLOWER]: Leader election lost for term 125. Reason: could not achieve majority
I20240503 08:12:08.630774 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 124 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:08.630864 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 124 FOLLOWER]: Advancing to term 125
I20240503 08:12:08.631511 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.631662 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 125 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.631880 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 125 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.631884 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 125 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.632004 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 125 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 125: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:08.632175 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 125 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.632267 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 125 FOLLOWER]: Leader election lost for term 125. Reason: could not achieve majority
I20240503 08:12:08.635666 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 125 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 124. Current term is 125. Ops: []
I20240503 08:12:08.635982 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 124.14293, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.657707 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 125 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 124. Current term is 125. Ops: []
I20240503 08:12:08.658033 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 124.14293, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.658947 26538 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 LEADER]: Stepping down as leader of term 124
I20240503 08:12:08.658994 26538 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:08.659077 26538 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14293, Committed index: 14293, Last appended: 124.14293, Last appended by leader: 14293, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.659168 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 124 FOLLOWER]: Advancing to term 125
I20240503 08:12:08.665442 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 125 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.665524 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 125 FOLLOWER]: Advancing to term 126
I20240503 08:12:08.666116 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.666271 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 126 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.666481 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 126 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.666481 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 126 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.666582 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 125 FOLLOWER]: Advancing to term 126
I20240503 08:12:08.666597 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 125 FOLLOWER]: Advancing to term 126
I20240503 08:12:08.667258 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 126.
I20240503 08:12:08.667258 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 126.
I20240503 08:12:08.667469 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:08.667560 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 126 FOLLOWER]: Leader election won for term 126
I20240503 08:12:08.667614 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 126 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.667668 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14293, Committed index: 14293, Last appended: 124.14293, Last appended by leader: 14293, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.668157 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 124 to 126, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 126 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:08.668288 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 124 current term: 126 is higher.
W20240503 08:12:08.676470 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 124 current term: 126 is higher.
I20240503 08:12:08.694540 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 126 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 124 index: 14293. Preceding OpId from leader: term: 126 index: 14294. (index mismatch)
I20240503 08:12:08.694849 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.697405 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 126 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.697479 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 126 FOLLOWER]: Advancing to term 127
I20240503 08:12:08.698099 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.698239 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 127 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.698433 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 127 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.698441 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 127 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.698700 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 127 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.698787 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 127 FOLLOWER]: Leader election lost for term 127. Reason: could not achieve majority
I20240503 08:12:08.703511 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 127 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 126. Current term is 127. Ops: []
I20240503 08:12:08.703743 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.728147 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 127 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 126. Current term is 127. Ops: [126.14294-126.14294]
I20240503 08:12:08.728503 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 124.14293, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.747814 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 127 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.747907 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 127 FOLLOWER]: Advancing to term 128
I20240503 08:12:08.748565 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.748724 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 128 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.748931 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 128 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.748970 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 128 candidate_status { last_received { term: 124 index: 14293 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.749171 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 128 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.749315 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Leader election lost for term 128. Reason: could not achieve majority
I20240503 08:12:08.750280 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 126 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:08.750331 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 126 FOLLOWER]: Advancing to term 127
I20240503 08:12:08.750963 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.751101 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 127 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.751283 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 127 candidate_status { last_received { term: 126 index: 14294 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.751288 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 127 candidate_status { last_received { term: 126 index: 14294 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.751368 26133 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 127. Current term is 128.
I20240503 08:12:08.751744 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 127 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 127. Current term is 128.
I20240503 08:12:08.751788 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 127 election: Cancelling election due to peer responding with higher term
I20240503 08:12:08.751888 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 127 FOLLOWER]: Advancing to term 128
I20240503 08:12:08.752420 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 128 FOLLOWER]: Leader election lost for term 127. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 127. Current term is 128.
I20240503 08:12:08.753111 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 126. Current term is 128. Ops: [126.14294-126.14294]
I20240503 08:12:08.753327 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 124.14293, Next index: 14294, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.753513 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 128 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 126. Current term is 128. Ops: []
I20240503 08:12:08.753741 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 126.14294, Next index: 14295, Last known committed idx: 14294, Time since last communication: 0.000s
I20240503 08:12:08.770084 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 126 LEADER]: Stepping down as leader of term 126
I20240503 08:12:08.770184 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 126 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:08.770264 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14294, Committed index: 14294, Last appended: 126.14294, Last appended by leader: 14294, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.770397 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 126 FOLLOWER]: Advancing to term 127
I20240503 08:12:08.788200 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 128 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.788311 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 128 FOLLOWER]: Advancing to term 129
I20240503 08:12:08.788964 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.789119 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 129 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.789330 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 129 candidate_status { last_received { term: 126 index: 14294 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.789394 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 128 FOLLOWER]: Advancing to term 129
I20240503 08:12:08.789346 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 129 candidate_status { last_received { term: 126 index: 14294 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.789458 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 127 FOLLOWER]: Advancing to term 129
I20240503 08:12:08.789971 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 129.
I20240503 08:12:08.789999 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 129.
I20240503 08:12:08.790189 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:08.790288 26538 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 129 FOLLOWER]: Leader election won for term 129
I20240503 08:12:08.790366 26538 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 129 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:08.790422 26538 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14294, Committed index: 14294, Last appended: 126.14294, Last appended by leader: 14294, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.790931 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 126 to 129, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 129 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:08.806337 26349 lock_manager.cc:494] Txn 5 has not acquired the partition lock after 1000ms
I20240503 08:12:08.816345 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 129 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 124 index: 14293. Preceding OpId from leader: term: 129 index: 14295. (index mismatch)
I20240503 08:12:08.816643 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14295, Last known committed idx: 14293, Time since last communication: 0.000s
I20240503 08:12:08.821316 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 129 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 126 index: 14294. Preceding OpId from leader: term: 129 index: 14295. (index mismatch)
I20240503 08:12:08.821606 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14295, Last known committed idx: 14294, Time since last communication: 0.000s
I20240503 08:12:08.847887 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 129 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:08.847997 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 129 FOLLOWER]: Advancing to term 130
I20240503 08:12:08.848670 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.848846 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 130 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.849066 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 130 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.849133 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 129. Current term is 130. Ops: []
I20240503 08:12:08.849066 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 130 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.849372 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 130 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.849457 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Leader election lost for term 130. Reason: could not achieve majority
I20240503 08:12:08.849472 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 129.14295, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.852552 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 129 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:08.852615 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 129 FOLLOWER]: Advancing to term 130
I20240503 08:12:08.853252 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.853413 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 130 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.853637 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 130 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.853646 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 130 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.853771 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 130: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:08.853897 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 130 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.854004 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 130 FOLLOWER]: Leader election lost for term 130. Reason: could not achieve majority
I20240503 08:12:08.854835 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 130 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 129. Current term is 130. Ops: []
I20240503 08:12:08.855051 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 129.14295, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.885864 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 129. Current term is 130. Ops: []
I20240503 08:12:08.886176 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 129.14295, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.888537 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 130 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 129. Current term is 130. Ops: []
I20240503 08:12:08.888873 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 129.14295, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.889987 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.890043 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 130 FOLLOWER]: Advancing to term 131
I20240503 08:12:08.891042 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 130 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.891103 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 130 FOLLOWER]: Advancing to term 131
I20240503 08:12:08.891836 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.892048 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 131 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.892113 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.892226 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 131 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.892273 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 131 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.892323 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 131: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:08.892244 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 131 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:08.892474 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 131 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:08.892657 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 131 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 131: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:08.892714 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 131 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.892827 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 131 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.892899 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 131 FOLLOWER]: Leader election lost for term 131. Reason: could not achieve majority
I20240503 08:12:08.892915 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 131 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:08.893020 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 131 FOLLOWER]: Leader election lost for term 131. Reason: could not achieve majority
I20240503 08:12:08.914748 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 131 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 129. Current term is 131. Ops: []
I20240503 08:12:08.915050 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 129.14295, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.923435 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 131 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 129. Current term is 131. Ops: []
I20240503 08:12:08.923754 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 129.14295, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.927152 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 129 LEADER]: Stepping down as leader of term 129
I20240503 08:12:08.927225 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 129 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:08.927301 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14295, Committed index: 14295, Last appended: 129.14295, Last appended by leader: 14295, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.927430 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 129 FOLLOWER]: Advancing to term 130
I20240503 08:12:08.936818 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 131 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:08.936939 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 131 FOLLOWER]: Advancing to term 132
I20240503 08:12:08.937640 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.937813 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 132 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:08.938238 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 132 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:08.938238 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 132 candidate_status { last_received { term: 129 index: 14295 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:08.938388 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 131 FOLLOWER]: Advancing to term 132
I20240503 08:12:08.938397 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 130 FOLLOWER]: Advancing to term 132
I20240503 08:12:08.939246 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 132.
I20240503 08:12:08.939246 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 132.
I20240503 08:12:08.939539 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:08.939631 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 132 FOLLOWER]: Leader election won for term 132
I20240503 08:12:08.939692 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 132 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:08.939735 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14295, Committed index: 14295, Last appended: 129.14295, Last appended by leader: 14295, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:08.940569 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 129 to 132, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 132 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:08.965957 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 132 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 129 index: 14295. Preceding OpId from leader: term: 132 index: 14296. (index mismatch)
I20240503 08:12:08.966270 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:08.968895 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 132 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 129 index: 14295. Preceding OpId from leader: term: 132 index: 14296. (index mismatch)
I20240503 08:12:08.969151 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14296, Last known committed idx: 14295, Time since last communication: 0.000s
I20240503 08:12:09.050961 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 132 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:09.051071 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 132 FOLLOWER]: Advancing to term 133
I20240503 08:12:09.051918 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.052126 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 133 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.052384 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 133 candidate_status { last_received { term: 132 index: 14296 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.052428 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 133 candidate_status { last_received { term: 132 index: 14296 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.052644 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.052752 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20240503 08:12:09.054731 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 133 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 132. Current term is 133. Ops: []
I20240503 08:12:09.055069 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 132.14296, Next index: 14297, Last known committed idx: 14296, Time since last communication: 0.000s
I20240503 08:12:09.069308 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 132 LEADER]: Stepping down as leader of term 132
I20240503 08:12:09.069407 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 132 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.069481 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14296, Committed index: 14296, Last appended: 132.14296, Last appended by leader: 14296, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.069605 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 132 FOLLOWER]: Advancing to term 133
I20240503 08:12:09.082588 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 132 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:09.082743 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 132 FOLLOWER]: Advancing to term 133
I20240503 08:12:09.083463 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.083652 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 133 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.083884 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 133 candidate_status { last_received { term: 132 index: 14296 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.083881 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 133 candidate_status { last_received { term: 132 index: 14296 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.083990 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 133: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:09.084570 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 133 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 133.
I20240503 08:12:09.084781 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 133 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:09.084888 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 133 FOLLOWER]: Leader election won for term 133
I20240503 08:12:09.084954 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 133 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.085007 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14296, Committed index: 14296, Last appended: 132.14296, Last appended by leader: 14296, Current term: 133, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.085536 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 132 to 133, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 133 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.094568 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 133 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.094709 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 133 FOLLOWER]: Advancing to term 134
I20240503 08:12:09.095367 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.095522 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 134 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.095752 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 134 candidate_status { last_received { term: 132 index: 14296 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.095755 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 134 candidate_status { last_received { term: 132 index: 14296 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.095858 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 133 FOLLOWER]: Advancing to term 134
I20240503 08:12:09.096462 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 134.
I20240503 08:12:09.096657 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.096767 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 FOLLOWER]: Leader election won for term 134
I20240503 08:12:09.096853 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.096910 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14296, Committed index: 14296, Last appended: 132.14296, Last appended by leader: 14296, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.097384 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 133 to 134, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 134 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:09.108525 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 133. Current term is 134. Ops: []
I20240503 08:12:09.108822 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14297, Last known committed idx: 14296, Time since last communication: 0.000s
I20240503 08:12:09.108930 26539 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 134.14297 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14296
I20240503 08:12:09.110276 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 134 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 133. Current term is 134. Ops: []
I20240503 08:12:09.110558 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14297, Last known committed idx: 14296, Time since last communication: 0.000s
I20240503 08:12:09.124516 26285 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 133 LEADER]: Stepping down as leader of term 133
I20240503 08:12:09.124599 26285 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 133 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.124651 26285 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14296, Committed index: 14296, Last appended: 133.14297, Last appended by leader: 14297, Current term: 133, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.124774 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 133 FOLLOWER]: Advancing to term 134
I20240503 08:12:09.125612 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 134 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 132 index: 14296. Preceding OpId from leader: term: 134 index: 14297. (index mismatch)
I20240503 08:12:09.125669 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 134 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 133 index: 14297. Preceding OpId from leader: term: 134 index: 14297. (term mismatch)
I20240503 08:12:09.125703 26285 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14296
I20240503 08:12:09.125748 26285 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted NO_OP operation due to leader change: 133.14297
I20240503 08:12:09.125787 26285 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 134 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:09.125941 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14297, Last known committed idx: 14296, Time since last communication: 0.000s
I20240503 08:12:09.126065 26531 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14297, Last known committed idx: 14296, Time since last communication: 0.000s
I20240503 08:12:09.157825 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 134 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:09.157938 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 134 FOLLOWER]: Advancing to term 135
I20240503 08:12:09.158706 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.158877 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 135 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.159089 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 135 candidate_status { last_received { term: 134 index: 14297 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.159240 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 135 candidate_status { last_received { term: 134 index: 14297 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.159507 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 135 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.159600 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 135 FOLLOWER]: Leader election lost for term 135. Reason: could not achieve majority
I20240503 08:12:09.160652 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 135 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 134. Current term is 135. Ops: []
I20240503 08:12:09.160867 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 134.14297, Next index: 14298, Last known committed idx: 14297, Time since last communication: 0.000s
I20240503 08:12:09.178257 26531 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 LEADER]: Stepping down as leader of term 134
I20240503 08:12:09.178369 26531 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.178447 26531 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14297, Committed index: 14297, Last appended: 134.14297, Last appended by leader: 14297, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.178587 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 134 FOLLOWER]: Advancing to term 135
I20240503 08:12:09.183959 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 134 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:09.184051 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 134 FOLLOWER]: Advancing to term 135
I20240503 08:12:09.184825 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.184994 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 135 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.185242 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 135 candidate_status { last_received { term: 134 index: 14297 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.185256 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 135 candidate_status { last_received { term: 134 index: 14297 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.185370 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 135: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:09.185921 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 135.
I20240503 08:12:09.186121 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.186225 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 135 FOLLOWER]: Leader election won for term 135
I20240503 08:12:09.186288 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 135 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.186348 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14297, Committed index: 14297, Last appended: 134.14297, Last appended by leader: 14297, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.186936 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 134 to 135, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 135 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.195808 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 135 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.195901 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 135 FOLLOWER]: Advancing to term 136
I20240503 08:12:09.196564 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.196820 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 136 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.196923 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 136 candidate_status { last_received { term: 134 index: 14297 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.196939 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 136 candidate_status { last_received { term: 134 index: 14297 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.197018 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 135 FOLLOWER]: Advancing to term 136
I20240503 08:12:09.197602 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 136.
I20240503 08:12:09.197851 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.197983 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 FOLLOWER]: Leader election won for term 136
I20240503 08:12:09.198052 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.198127 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14297, Committed index: 14297, Last appended: 134.14297, Last appended by leader: 14297, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.198560 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 135 to 136, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 136 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.215659 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 135. Current term is 136. Ops: []
I20240503 08:12:09.215971 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14298, Last known committed idx: 14297, Time since last communication: 0.000s
I20240503 08:12:09.216063 26517 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 136.14298 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14297
I20240503 08:12:09.217183 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 136 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 135. Current term is 136. Ops: []
I20240503 08:12:09.217414 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14298, Last known committed idx: 14297, Time since last communication: 0.000s
I20240503 08:12:09.227169 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 136 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 134 index: 14297. Preceding OpId from leader: term: 136 index: 14298. (index mismatch)
I20240503 08:12:09.227434 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14298, Last known committed idx: 14297, Time since last communication: 0.000s
I20240503 08:12:09.234239 26285 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 135 LEADER]: Stepping down as leader of term 135
I20240503 08:12:09.234333 26285 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 135 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.234405 26285 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14297, Committed index: 14297, Last appended: 135.14298, Last appended by leader: 14298, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.234532 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 135 FOLLOWER]: Advancing to term 136
I20240503 08:12:09.235217 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 136 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 135 index: 14298. Preceding OpId from leader: term: 136 index: 14298. (term mismatch)
I20240503 08:12:09.235260 26285 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14297
I20240503 08:12:09.235280 26285 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted NO_OP operation due to leader change: 135.14298
I20240503 08:12:09.235301 26285 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 136 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:09.235607 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14298, Last known committed idx: 14297, Time since last communication: 0.000s
I20240503 08:12:09.266665 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 136 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:09.266770 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 136 FOLLOWER]: Advancing to term 137
I20240503 08:12:09.267472 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.267660 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 136. Current term is 137. Ops: []
I20240503 08:12:09.267697 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 137 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.267853 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 137 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.267866 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 137 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.268075 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.268076 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 136.14298, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.268191 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I20240503 08:12:09.284912 26531 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 136 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:09.285012 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 136 FOLLOWER]: Advancing to term 137
I20240503 08:12:09.285657 26531 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.285831 26531 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 137 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.286001 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 137 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.286017 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 137 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.286103 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 137: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:09.286257 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.286370 26531 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I20240503 08:12:09.291530 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 137 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 136. Current term is 137. Ops: []
I20240503 08:12:09.291790 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 136.14298, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.303187 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 136. Current term is 137. Ops: []
I20240503 08:12:09.303552 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 136.14298, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.311573 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.311667 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 137 FOLLOWER]: Advancing to term 138
I20240503 08:12:09.312369 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.312517 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 138 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.312743 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 138 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.312758 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 138 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.312870 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 137 FOLLOWER]: Advancing to term 138
I20240503 08:12:09.313463 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 138.
I20240503 08:12:09.313656 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.313762 26538 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 138 FOLLOWER]: Leader election won for term 138
I20240503 08:12:09.313827 26538 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 138 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.313884 26538 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14298, Committed index: 14298, Last appended: 136.14298, Last appended by leader: 14298, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.314374 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 136 to 138, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 138 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.317775 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 LEADER]: Stepping down as leader of term 136
I20240503 08:12:09.317849 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.317932 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14298, Committed index: 14298, Last appended: 136.14298, Last appended by leader: 14298, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.318023 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 136 FOLLOWER]: Advancing to term 137
I20240503 08:12:09.337924 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 137 FOLLOWER]: Advancing to term 138
I20240503 08:12:09.338714 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 138 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 136 index: 14298. Preceding OpId from leader: term: 138 index: 14299. (index mismatch)
I20240503 08:12:09.339059 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.347190 26531 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 138 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.347285 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 138 FOLLOWER]: Advancing to term 139
I20240503 08:12:09.348096 26531 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.348294 26531 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 139 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.348265 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 138. Current term is 139. Ops: []
I20240503 08:12:09.348479 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 139 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.348479 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 139 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.348691 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.348757 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 139 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.348858 26531 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Leader election lost for term 139. Reason: could not achieve majority
I20240503 08:12:09.366920 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 138. Current term is 139. Ops: [138.14299-138.14299]
I20240503 08:12:09.367236 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 136.14298, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.392119 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 138. Current term is 139. Ops: [138.14299-138.14299]
I20240503 08:12:09.392429 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 136.14298, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.399932 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 138 LEADER]: Stepping down as leader of term 138
I20240503 08:12:09.400012 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 138 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.400094 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14299, Committed index: 14299, Last appended: 138.14299, Last appended by leader: 14299, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.400203 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 138 FOLLOWER]: Advancing to term 139
I20240503 08:12:09.400274 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 138 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:09.400315 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 138 FOLLOWER]: Advancing to term 139
I20240503 08:12:09.401031 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.401186 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 139 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.401335 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 139 candidate_status { last_received { term: 138 index: 14299 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.401430 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 139 candidate_status { last_received { term: 138 index: 14299 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.401532 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 139: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:09.402266 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 139.
I20240503 08:12:09.402487 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:09.402572 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 139 FOLLOWER]: Leader election won for term 139
I20240503 08:12:09.402653 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 139 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.402702 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14299, Committed index: 14299, Last appended: 138.14299, Last appended by leader: 14299, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.403124 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 138 to 139, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 139 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.412901 26531 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.412997 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 139 FOLLOWER]: Advancing to term 140
I20240503 08:12:09.413676 26531 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.413834 26531 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 140 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.414057 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 140 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.414125 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 139 FOLLOWER]: Advancing to term 140
I20240503 08:12:09.414060 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 140 candidate_status { last_received { term: 136 index: 14298 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.414733 26285 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 140 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 140 because replica has last-logged OpId of term: 138 index: 14299, which is greater than that of the candidate, which has last-logged OpId of term: 136 index: 14298.
I20240503 08:12:09.414943 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 140 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.415033 26531 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 140 FOLLOWER]: Leader election lost for term 140. Reason: could not achieve majority
I20240503 08:12:09.428804 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 140 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 139. Current term is 140. Ops: []
I20240503 08:12:09.429208 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14300, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.434979 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 140 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 139. Current term is 140. Ops: []
I20240503 08:12:09.435297 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14300, Last known committed idx: 14299, Time since last communication: 0.000s
I20240503 08:12:09.446650 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 140 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.446753 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 140 FOLLOWER]: Advancing to term 141
I20240503 08:12:09.447455 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.447654 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 141 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.447973 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 141 candidate_status { last_received { term: 138 index: 14299 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.447978 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 141 candidate_status { last_received { term: 138 index: 14299 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.448117 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 140 FOLLOWER]: Advancing to term 141
I20240503 08:12:09.448697 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 141.
I20240503 08:12:09.448928 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.449019 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 141 FOLLOWER]: Leader election won for term 141
I20240503 08:12:09.449070 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 141 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.449111 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14299, Committed index: 14299, Last appended: 138.14299, Last appended by leader: 14299, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.449752 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 139 to 141, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 141 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.451718 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 141 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 139. Current term is 141. Ops: [138.14299-139.14300]
I20240503 08:12:09.452019 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 136.14298, Next index: 14299, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.464483 26502 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 139 LEADER]: Stepping down as leader of term 139
I20240503 08:12:09.464599 26502 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.464668 26502 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14299, Committed index: 14299, Last appended: 139.14300, Last appended by leader: 14300, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.464780 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 139 FOLLOWER]: Advancing to term 140
I20240503 08:12:09.483175 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 141 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 136 index: 14298. Preceding OpId from leader: term: 141 index: 14300. (index mismatch)
I20240503 08:12:09.483527 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14300, Last known committed idx: 14298, Time since last communication: 0.000s
I20240503 08:12:09.485138 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 140 FOLLOWER]: Advancing to term 141
I20240503 08:12:09.485826 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 141 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 139 index: 14300. Preceding OpId from leader: term: 141 index: 14300. (term mismatch)
I20240503 08:12:09.485863 26133 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14299
I20240503 08:12:09.485883 26133 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 139.14300
I20240503 08:12:09.485906 26133 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 141 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:09.486168 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14300, Last known committed idx: 14299, Time since last communication: 0.000s
I20240503 08:12:09.515779 26531 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 141 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:09.515899 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 141 FOLLOWER]: Advancing to term 142
I20240503 08:12:09.516815 26531 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.517017 26531 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 142 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.517264 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 142 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.517269 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 142 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.517427 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 141 FOLLOWER]: Advancing to term 142
I20240503 08:12:09.517838 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 141. Current term is 142. Ops: []
I20240503 08:12:09.518038 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 141.14300, Next index: 14301, Last known committed idx: 14300, Time since last communication: 0.000s
I20240503 08:12:09.518190 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 142.
I20240503 08:12:09.518253 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 142 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.518334 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 142 FOLLOWER]: Advancing to term 143
I20240503 08:12:09.518366 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.518457 26531 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 FOLLOWER]: Leader election won for term 142
I20240503 08:12:09.518502 26531 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.518551 26531 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14300, Committed index: 14300, Last appended: 141.14300, Last appended by leader: 14300, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.518985 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.519039 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 141 to 142, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 142 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:09.519083 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 143 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 141. Current term is 143. Ops: []
I20240503 08:12:09.519116 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 143 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.519232 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 143 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.519255 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 141.14300, Next index: 14301, Last known committed idx: 14300, Time since last communication: 0.000s
I20240503 08:12:09.519496 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 143 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.519712 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 143 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.519825 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 143 FOLLOWER]: Leader election lost for term 143. Reason: could not achieve majority
I20240503 08:12:09.542476 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 143 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 142. Current term is 143. Ops: []
I20240503 08:12:09.542841 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14301, Last known committed idx: 14300, Time since last communication: 0.000s
I20240503 08:12:09.549396 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 141. Current term is 142. Ops: []
I20240503 08:12:09.549716 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 141.14300, Next index: 14301, Last known committed idx: 14300, Time since last communication: 0.000s
I20240503 08:12:09.549823 26517 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 142.14301 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14300
I20240503 08:12:09.552382 26285 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 141 LEADER]: Stepping down as leader of term 141
I20240503 08:12:09.552448 26285 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 141 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.552524 26285 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14300, Committed index: 14300, Last appended: 141.14300, Last appended by leader: 14300, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.552634 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 141 FOLLOWER]: Advancing to term 142
I20240503 08:12:09.553269 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 142 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 141 index: 14300. Preceding OpId from leader: term: 142 index: 14301. (index mismatch)
I20240503 08:12:09.553604 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14301, Last known committed idx: 14300, Time since last communication: 0.000s
I20240503 08:12:09.558076 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 143 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.558159 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 143 FOLLOWER]: Advancing to term 144
I20240503 08:12:09.558873 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.559146 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 144 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.559350 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 144 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.559379 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 144 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.559602 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 144 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.559739 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 144 FOLLOWER]: Leader election lost for term 144. Reason: could not achieve majority
I20240503 08:12:09.576119 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 144 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 142. Current term is 144. Ops: [142.14301-142.14301]
I20240503 08:12:09.576508 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 141.14300, Next index: 14301, Last known committed idx: 14300, Time since last communication: 0.000s
I20240503 08:12:09.582093 26531 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 LEADER]: Stepping down as leader of term 142
I20240503 08:12:09.582190 26531 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.582262 26531 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14301, Committed index: 14301, Last appended: 142.14301, Last appended by leader: 14301, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.582391 26531 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 142 FOLLOWER]: Advancing to term 143
I20240503 08:12:09.597452 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 144 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.597563 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 144 FOLLOWER]: Advancing to term 145
I20240503 08:12:09.598268 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.598420 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 145 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.598672 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 145 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.598685 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 145 candidate_status { last_received { term: 141 index: 14300 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.598784 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 143 FOLLOWER]: Advancing to term 145
I20240503 08:12:09.599395 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 145 because replica has last-logged OpId of term: 142 index: 14301, which is greater than that of the candidate, which has last-logged OpId of term: 141 index: 14300.
I20240503 08:12:09.599606 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 145 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.599699 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 145 FOLLOWER]: Leader election lost for term 145. Reason: could not achieve majority
I20240503 08:12:09.611176 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 142 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:09.611303 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 142 FOLLOWER]: Advancing to term 143
I20240503 08:12:09.611831 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 145 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.611900 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 145 FOLLOWER]: Advancing to term 146
I20240503 08:12:09.612401 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.612548 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.612942 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 146 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.612653 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 143 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.613068 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 143 candidate_status { last_received { term: 142 index: 14301 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.613122 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 146 candidate_status { last_received { term: 142 index: 14301 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.613168 26133 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 143. Current term is 145.
I20240503 08:12:09.613168 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 146 candidate_status { last_received { term: 142 index: 14301 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.613278 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 143 candidate_status { last_received { term: 142 index: 14301 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.613319 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 143 FOLLOWER]: Advancing to term 146
I20240503 08:12:09.613329 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 143. Current term is 146.
I20240503 08:12:09.613395 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 143 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 143. Current term is 145.
I20240503 08:12:09.613449 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 143 election: Cancelling election due to peer responding with higher term
I20240503 08:12:09.613556 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 143 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 143. Current term is 146.
I20240503 08:12:09.613977 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 146.
I20240503 08:12:09.614070 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 146 FOLLOWER]: Leader election lost for term 143. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 143. Current term is 145.
I20240503 08:12:09.614256 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.614388 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 FOLLOWER]: Leader election won for term 146
I20240503 08:12:09.614463 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.614500 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14300, Committed index: 14300, Last appended: 142.14301, Last appended by leader: 14301, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.615110 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 142 to 146. New cstate: current_term: 146 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:09.643654 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 146 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 142 index: 14301. Preceding OpId from leader: term: 146 index: 14302. (index mismatch)
I20240503 08:12:09.644009 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14302, Last known committed idx: 14301, Time since last communication: 0.000s
I20240503 08:12:09.644419 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 145 FOLLOWER]: Advancing to term 146
I20240503 08:12:09.645368 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 146 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 141 index: 14300. Preceding OpId from leader: term: 146 index: 14302. (index mismatch)
I20240503 08:12:09.645572 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14302, Last known committed idx: 14300, Time since last communication: 0.000s
W20240503 08:12:09.671577 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 144 current term: 146 is higher.
I20240503 08:12:09.750692 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 146 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:09.750794 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 146 FOLLOWER]: Advancing to term 147
I20240503 08:12:09.751511 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.751668 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 147 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.751875 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 147 candidate_status { last_received { term: 146 index: 14302 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.751892 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 147 candidate_status { last_received { term: 146 index: 14302 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.752136 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 147 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.752262 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 147 FOLLOWER]: Leader election lost for term 147. Reason: could not achieve majority
I20240503 08:12:09.754513 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 147 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 146. Current term is 147. Ops: []
I20240503 08:12:09.754849 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 146.14302, Next index: 14303, Last known committed idx: 14302, Time since last communication: 0.000s
I20240503 08:12:09.761096 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 LEADER]: Stepping down as leader of term 146
I20240503 08:12:09.761183 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.761248 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14302, Committed index: 14302, Last appended: 146.14302, Last appended by leader: 14302, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.761368 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 146 FOLLOWER]: Advancing to term 147
I20240503 08:12:09.779489 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 146 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:09.779592 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 146 FOLLOWER]: Advancing to term 147
I20240503 08:12:09.780259 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.780436 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 147 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.780668 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 147 candidate_status { last_received { term: 146 index: 14302 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.780685 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 147 candidate_status { last_received { term: 146 index: 14302 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.780791 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 147 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 147: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:09.781315 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 147.
I20240503 08:12:09.781550 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.781642 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 147 FOLLOWER]: Leader election won for term 147
I20240503 08:12:09.781698 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 147 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.781740 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14302, Committed index: 14302, Last appended: 146.14302, Last appended by leader: 14302, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.782305 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 146 to 147, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 147 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.791463 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 147 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.791570 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 147 FOLLOWER]: Advancing to term 148
I20240503 08:12:09.792281 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.792477 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 148 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.792661 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 148 candidate_status { last_received { term: 146 index: 14302 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.792665 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 148 candidate_status { last_received { term: 146 index: 14302 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.792778 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 147 FOLLOWER]: Advancing to term 148
I20240503 08:12:09.793445 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 148.
I20240503 08:12:09.793648 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.793792 26538 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 FOLLOWER]: Leader election won for term 148
I20240503 08:12:09.793864 26538 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.793931 26538 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14302, Committed index: 14302, Last appended: 146.14302, Last appended by leader: 14302, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.794376 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 147 to 148, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 148 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:09.807437 26351 lock_manager.cc:494] Txn 5 has not acquired the partition lock after 1000ms
I20240503 08:12:09.808566 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 148 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 147. Current term is 148. Ops: []
I20240503 08:12:09.808569 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 147 LEADER]: Stepping down as leader of term 147
I20240503 08:12:09.808714 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 147. Current term is 148. Ops: []
I20240503 08:12:09.808727 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 147 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.808774 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 148 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 146 index: 14302. Preceding OpId from leader: term: 148 index: 14304. (index mismatch)
I20240503 08:12:09.808856 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14302, Committed index: 14302, Last appended: 147.14304, Last appended by leader: 14304, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.808962 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 147 FOLLOWER]: Advancing to term 148
W20240503 08:12:09.808952 26516 consensus_queue.cc:1159] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" responder_term: 148 status { last_received { term: 148 index: 14304 } last_committed_idx: 14302 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 147. Current term is 148. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240503 08:12:09.809907 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 148 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 147 index: 14304. Preceding OpId from leader: term: 148 index: 14304. (term mismatch)
I20240503 08:12:09.809959 26133 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14303
I20240503 08:12:09.809981 26133 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 147.14304
I20240503 08:12:09.810027 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14303, Last known committed idx: 14302, Time since last communication: 0.000s
I20240503 08:12:09.810230 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14303, Last known committed idx: 14302, Time since last communication: 0.000s
I20240503 08:12:09.810292 26539 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 147.14303 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14302
I20240503 08:12:09.810508 26133 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14302
I20240503 08:12:09.810554 26133 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 147.14303
I20240503 08:12:09.810593 26133 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 148 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:09.841377 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 148 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:09.841495 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 148 FOLLOWER]: Advancing to term 149
I20240503 08:12:09.842157 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.842335 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 149 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.842661 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 149 candidate_status { last_received { term: 148 index: 14304 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.842669 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 149 candidate_status { last_received { term: 148 index: 14304 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.843241 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 149 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.843351 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 149 FOLLOWER]: Leader election lost for term 149. Reason: could not achieve majority
I20240503 08:12:09.844117 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 149 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 148. Current term is 149. Ops: []
I20240503 08:12:09.844383 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 148.14304, Next index: 14305, Last known committed idx: 14304, Time since last communication: 0.000s
I20240503 08:12:09.868842 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 148 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:09.868944 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 148 FOLLOWER]: Advancing to term 149
I20240503 08:12:09.869599 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.869766 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 149 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.869988 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 149 candidate_status { last_received { term: 148 index: 14304 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.870064 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 149 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 149: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:09.869999 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 149 candidate_status { last_received { term: 148 index: 14304 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.870262 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 149 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.870371 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 149 FOLLOWER]: Leader election lost for term 149. Reason: could not achieve majority
I20240503 08:12:09.873894 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 149 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.873968 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 149 FOLLOWER]: Advancing to term 150
I20240503 08:12:09.874481 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 149 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 148. Current term is 149. Ops: []
I20240503 08:12:09.874719 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 148.14304, Next index: 14305, Last known committed idx: 14304, Time since last communication: 0.000s
I20240503 08:12:09.874873 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.875034 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 150 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.875164 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 150 candidate_status { last_received { term: 148 index: 14304 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.875176 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 150 candidate_status { last_received { term: 148 index: 14304 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.875254 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 149 FOLLOWER]: Advancing to term 150
I20240503 08:12:09.875859 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 150.
I20240503 08:12:09.876044 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.876149 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 FOLLOWER]: Leader election won for term 150
I20240503 08:12:09.876207 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.876263 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14304, Committed index: 14304, Last appended: 148.14304, Last appended by leader: 14304, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.876725 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 148 to 150, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 150 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:09.878154 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 148. Current term is 150. Ops: []
I20240503 08:12:09.878417 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 148.14304, Next index: 14305, Last known committed idx: 14304, Time since last communication: 0.000s
I20240503 08:12:09.878499 26539 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 150.14305 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14304
I20240503 08:12:09.901014 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 150 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 148 index: 14304. Preceding OpId from leader: term: 150 index: 14305. (index mismatch)
I20240503 08:12:09.901336 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14305, Last known committed idx: 14304, Time since last communication: 0.000s
I20240503 08:12:09.903366 26285 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 LEADER]: Stepping down as leader of term 148
I20240503 08:12:09.903425 26285 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:09.903474 26285 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14304, Committed index: 14304, Last appended: 148.14304, Last appended by leader: 14304, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.903573 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 148 FOLLOWER]: Advancing to term 150
I20240503 08:12:09.904220 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 150 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 148 index: 14304. Preceding OpId from leader: term: 150 index: 14305. (index mismatch)
I20240503 08:12:09.904443 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14305, Last known committed idx: 14304, Time since last communication: 0.000s
I20240503 08:12:09.932626 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 150 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:09.932739 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 150 FOLLOWER]: Advancing to term 151
I20240503 08:12:09.933411 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.933575 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 151 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.933836 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 151 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.933836 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 151 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.934108 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 151 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:09.934221 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 151 FOLLOWER]: Leader election lost for term 151. Reason: could not achieve majority
I20240503 08:12:09.935364 26538 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 150 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:09.935432 26538 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 150 FOLLOWER]: Advancing to term 151
I20240503 08:12:09.936061 26538 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.936216 26538 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 151 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:09.936456 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 151 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.936504 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 151 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:09.936578 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 151 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 150. Current term is 151. Ops: []
I20240503 08:12:09.936722 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 151 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:09.936798 26538 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 151 FOLLOWER]: Leader election lost for term 151. Reason: could not achieve majority
I20240503 08:12:09.936807 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 150.14305, Next index: 14306, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:09.940266 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 151 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 150. Current term is 151. Ops: []
I20240503 08:12:09.940654 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 150.14305, Next index: 14306, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:09.961477 26516 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 LEADER]: Stepping down as leader of term 150
I20240503 08:12:09.961582 26516 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:09.961668 26516 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14305, Committed index: 14305, Last appended: 150.14305, Last appended by leader: 14305, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.961803 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 150 FOLLOWER]: Advancing to term 151
I20240503 08:12:09.974007 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 151 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:09.974123 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 151 FOLLOWER]: Advancing to term 152
I20240503 08:12:09.974840 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.975015 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 152 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:09.975230 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 152 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:09.975293 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 151 FOLLOWER]: Advancing to term 152
I20240503 08:12:09.975239 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 152 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:09.975332 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 151 FOLLOWER]: Advancing to term 152
I20240503 08:12:09.975811 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 152.
I20240503 08:12:09.975875 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 152.
I20240503 08:12:09.976042 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:09.976141 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 152 FOLLOWER]: Leader election won for term 152
I20240503 08:12:09.976200 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 152 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:09.976258 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14305, Committed index: 14305, Last appended: 150.14305, Last appended by leader: 14305, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:09.976699 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 150 to 152, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 152 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:10.007400 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 152 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 150 index: 14305. Preceding OpId from leader: term: 152 index: 14306. (index mismatch)
I20240503 08:12:10.007673 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14306, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:10.009371 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 152 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.009438 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 152 FOLLOWER]: Advancing to term 153
I20240503 08:12:10.010027 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.010156 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 153 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.010331 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 153 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.010371 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 153 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.010563 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 153 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.010687 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Leader election lost for term 153. Reason: could not achieve majority
I20240503 08:12:10.011027 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 152. Current term is 153. Ops: []
I20240503 08:12:10.011194 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14306, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:10.011706 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 152. Current term is 153. Ops: [152.14306-152.14306]
I20240503 08:12:10.011910 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 150.14305, Next index: 14306, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:10.045326 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 152. Current term is 153. Ops: [152.14306-152.14306]
I20240503 08:12:10.045737 26531 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 150.14305, Next index: 14306, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:10.054309 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.054417 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 153 FOLLOWER]: Advancing to term 154
I20240503 08:12:10.055124 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.055274 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 154 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.055501 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 154 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.055511 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 154 candidate_status { last_received { term: 150 index: 14305 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.055739 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 154 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.055842 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 154 FOLLOWER]: Leader election lost for term 154. Reason: could not achieve majority
I20240503 08:12:10.056741 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 152 LEADER]: Stepping down as leader of term 152
I20240503 08:12:10.056785 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 152 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:10.056838 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14306, Committed index: 14306, Last appended: 152.14306, Last appended by leader: 14306, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.056924 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 152 FOLLOWER]: Advancing to term 153
I20240503 08:12:10.067667 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 152 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:10.067757 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 152 FOLLOWER]: Advancing to term 153
I20240503 08:12:10.068397 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.068610 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 153 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.068811 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 153 candidate_status { last_received { term: 152 index: 14306 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.068811 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 153 candidate_status { last_received { term: 152 index: 14306 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.068928 26133 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 154 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 153. Current term is 154.
I20240503 08:12:10.069109 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 153 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 154 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 153. Current term is 154.
I20240503 08:12:10.069169 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 153 election: Cancelling election due to peer responding with higher term
I20240503 08:12:10.069236 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 153 FOLLOWER]: Advancing to term 154
I20240503 08:12:10.069470 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 153.
I20240503 08:12:10.069813 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 154 FOLLOWER]: Leader election lost for term 153. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 154 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 153. Current term is 154.
I20240503 08:12:10.106530 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 154 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.106662 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 154 FOLLOWER]: Advancing to term 155
I20240503 08:12:10.107419 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.107590 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 155 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.107805 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 155 candidate_status { last_received { term: 152 index: 14306 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.107823 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 155 candidate_status { last_received { term: 152 index: 14306 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.107890 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 153 FOLLOWER]: Advancing to term 155
I20240503 08:12:10.107897 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 154 FOLLOWER]: Advancing to term 155
I20240503 08:12:10.108500 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 155.
I20240503 08:12:10.108506 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 155.
I20240503 08:12:10.108719 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:10.108812 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 155 FOLLOWER]: Leader election won for term 155
I20240503 08:12:10.108862 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 155 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:10.108927 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14306, Committed index: 14306, Last appended: 152.14306, Last appended by leader: 14306, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.109377 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 152 to 155, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 155 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.138130 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 155 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 150 index: 14305. Preceding OpId from leader: term: 155 index: 14307. (index mismatch)
I20240503 08:12:10.138515 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14307, Last known committed idx: 14305, Time since last communication: 0.000s
I20240503 08:12:10.140954 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 155 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 152 index: 14306. Preceding OpId from leader: term: 155 index: 14307. (index mismatch)
I20240503 08:12:10.141260 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14307, Last known committed idx: 14306, Time since last communication: 0.000s
I20240503 08:12:10.170500 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 155 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:10.170595 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 155 FOLLOWER]: Advancing to term 156
I20240503 08:12:10.171365 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.171582 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 156 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.171655 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 156 candidate_status { last_received { term: 155 index: 14307 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.171670 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 156 candidate_status { last_received { term: 155 index: 14307 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.171921 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.172030 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20240503 08:12:10.177044 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 156 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 155. Current term is 156. Ops: []
I20240503 08:12:10.177312 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 155.14307, Next index: 14308, Last known committed idx: 14307, Time since last communication: 0.000s
I20240503 08:12:10.207628 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 156 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 155. Current term is 156. Ops: []
I20240503 08:12:10.207911 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 155.14307, Next index: 14308, Last known committed idx: 14307, Time since last communication: 0.000s
I20240503 08:12:10.213407 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 155 LEADER]: Stepping down as leader of term 155
I20240503 08:12:10.213506 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 155 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:10.213580 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14307, Committed index: 14307, Last appended: 155.14307, Last appended by leader: 14307, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.213704 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 155 FOLLOWER]: Advancing to term 156
I20240503 08:12:10.216681 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 156 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.216748 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 156 FOLLOWER]: Advancing to term 157
I20240503 08:12:10.217345 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.217592 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 157 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.217690 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 157 candidate_status { last_received { term: 155 index: 14307 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.217690 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 157 candidate_status { last_received { term: 155 index: 14307 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.217797 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 156 FOLLOWER]: Advancing to term 157
I20240503 08:12:10.218600 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 157.
I20240503 08:12:10.219017 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:10.219177 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 157 FOLLOWER]: Leader election won for term 157
I20240503 08:12:10.219251 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 157 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:10.219357 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14307, Committed index: 14307, Last appended: 155.14307, Last appended by leader: 14307, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.220288 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 155 to 157, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 157 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.227171 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 155 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:10.227267 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 155 FOLLOWER]: Advancing to term 156
I20240503 08:12:10.228189 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.228375 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 156 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.228567 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 156 candidate_status { last_received { term: 155 index: 14307 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.228567 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 156 candidate_status { last_received { term: 155 index: 14307 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.228680 26285 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 156. Current term is 157.
I20240503 08:12:10.228824 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 156 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 156. Current term is 157.
I20240503 08:12:10.228870 26173 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 156 election: Cancelling election due to peer responding with higher term
I20240503 08:12:10.228942 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 156 FOLLOWER]: Advancing to term 157
I20240503 08:12:10.229462 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 157 FOLLOWER]: Leader election lost for term 156. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 156. Current term is 157.
I20240503 08:12:10.249779 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 155 index: 14307. Preceding OpId from leader: term: 157 index: 14308. (index mismatch)
I20240503 08:12:10.250160 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14308, Last known committed idx: 14307, Time since last communication: 0.000s
I20240503 08:12:10.253875 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 157 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 155 index: 14307. Preceding OpId from leader: term: 157 index: 14308. (index mismatch)
I20240503 08:12:10.254355 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14308, Last known committed idx: 14307, Time since last communication: 0.000s
I20240503 08:12:10.307612 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 157 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:10.307713 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 157 FOLLOWER]: Advancing to term 158
I20240503 08:12:10.308068 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:10.308128 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 157 FOLLOWER]: Advancing to term 158
I20240503 08:12:10.308465 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.308621 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 158 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.308745 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.308888 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 158 candidate_status { last_received { term: 157 index: 14308 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.308888 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 158 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.308976 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 158: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:10.309023 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 158 candidate_status { last_received { term: 157 index: 14308 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.309039 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 158 candidate_status { last_received { term: 157 index: 14308 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.309005 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 158 candidate_status { last_received { term: 157 index: 14308 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.309155 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 158: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:10.309187 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.309260 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:10.309286 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20240503 08:12:10.309353 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20240503 08:12:10.313796 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 158 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 157. Current term is 158. Ops: []
I20240503 08:12:10.313828 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 158 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 157. Current term is 158. Ops: []
I20240503 08:12:10.314124 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 157.14308, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.314225 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 157.14308, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.340052 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 158 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.340157 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 158 FOLLOWER]: Advancing to term 159
I20240503 08:12:10.340888 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.341173 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 159 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.341459 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 159 candidate_status { last_received { term: 157 index: 14308 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.341459 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 159 candidate_status { last_received { term: 157 index: 14308 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.341575 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 158 FOLLOWER]: Advancing to term 159
I20240503 08:12:10.342182 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 159.
I20240503 08:12:10.342415 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:10.342526 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 FOLLOWER]: Leader election won for term 159
I20240503 08:12:10.342586 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:10.342658 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14308, Committed index: 14308, Last appended: 157.14308, Last appended by leader: 14308, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.343325 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 157 to 159, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 159 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.348537 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 157. Current term is 159. Ops: []
I20240503 08:12:10.349109 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 157.14308, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.349336 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 159 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 157. Current term is 159. Ops: []
I20240503 08:12:10.349366 26515 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 159.14309 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14308
I20240503 08:12:10.349790 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 157.14308, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.370640 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 157 LEADER]: Stepping down as leader of term 157
I20240503 08:12:10.370666 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 157. Current term is 159. Ops: []
I20240503 08:12:10.370733 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 157 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:10.370802 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14308, Committed index: 14308, Last appended: 157.14308, Last appended by leader: 14308, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.370925 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 157.14308, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.371034 26502 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 159.14309 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14308
I20240503 08:12:10.371074 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 159 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 157 index: 14308. Preceding OpId from leader: term: 159 index: 14309. (index mismatch)
I20240503 08:12:10.371212 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 157 FOLLOWER]: Advancing to term 159
I20240503 08:12:10.371269 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.372246 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 159 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 157 index: 14308. Preceding OpId from leader: term: 159 index: 14309. (index mismatch)
I20240503 08:12:10.372459 26538 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14309, Last known committed idx: 14308, Time since last communication: 0.000s
I20240503 08:12:10.425571 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 159 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:10.425662 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 159 FOLLOWER]: Advancing to term 160
I20240503 08:12:10.426299 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.426447 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 160 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.426661 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 160 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.426666 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 160 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.426927 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 160 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.427035 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 160 FOLLOWER]: Leader election lost for term 160. Reason: could not achieve majority
I20240503 08:12:10.427786 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 160 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 159. Current term is 160. Ops: []
I20240503 08:12:10.428059 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 159.14309, Next index: 14310, Last known committed idx: 14309, Time since last communication: 0.000s
I20240503 08:12:10.454888 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 159 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:10.455027 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 159 FOLLOWER]: Advancing to term 160
I20240503 08:12:10.455997 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.456171 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 160 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.456351 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 160 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.456441 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 160 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 160: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:10.456521 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 160 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.456702 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 160 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.456820 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 160 FOLLOWER]: Leader election lost for term 160. Reason: could not achieve majority
I20240503 08:12:10.457595 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 160 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.457641 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 160 FOLLOWER]: Advancing to term 161
I20240503 08:12:10.458212 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.458370 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 161 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.458537 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 161 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.458596 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 160 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 159. Current term is 160. Ops: []
I20240503 08:12:10.458555 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 161 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.458840 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 159.14309, Next index: 14310, Last known committed idx: 14309, Time since last communication: 0.000s
I20240503 08:12:10.458842 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 161 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.459115 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 161 FOLLOWER]: Leader election lost for term 161. Reason: could not achieve majority
I20240503 08:12:10.463495 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 161 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 159. Current term is 161. Ops: []
I20240503 08:12:10.463822 26538 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 159.14309, Next index: 14310, Last known committed idx: 14309, Time since last communication: 0.000s
I20240503 08:12:10.474529 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 LEADER]: Stepping down as leader of term 159
I20240503 08:12:10.474653 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:10.474761 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14309, Committed index: 14309, Last appended: 159.14309, Last appended by leader: 14309, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.474884 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 159 FOLLOWER]: Advancing to term 160
I20240503 08:12:10.493747 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 161 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.493845 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 161 FOLLOWER]: Advancing to term 162
I20240503 08:12:10.494475 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.494673 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 162 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.494884 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 162 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.494949 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 160 FOLLOWER]: Advancing to term 162
I20240503 08:12:10.494925 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 162 candidate_status { last_received { term: 159 index: 14309 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.495074 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 160 FOLLOWER]: Advancing to term 162
I20240503 08:12:10.495600 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 162 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 162.
I20240503 08:12:10.495643 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 162 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 162.
I20240503 08:12:10.495810 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 162 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:10.495908 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 162 FOLLOWER]: Leader election won for term 162
I20240503 08:12:10.495967 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 162 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:10.496038 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14309, Committed index: 14309, Last appended: 159.14309, Last appended by leader: 14309, Current term: 162, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.496553 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 159 to 162, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 162 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.518982 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 162 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 159 index: 14309. Preceding OpId from leader: term: 162 index: 14310. (index mismatch)
I20240503 08:12:10.519344 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14310, Last known committed idx: 14309, Time since last communication: 0.000s
I20240503 08:12:10.531316 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 162 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 159 index: 14309. Preceding OpId from leader: term: 162 index: 14310. (index mismatch)
I20240503 08:12:10.531615 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14310, Last known committed idx: 14309, Time since last communication: 0.000s
I20240503 08:12:10.551316 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 162 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:10.551414 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 162 FOLLOWER]: Advancing to term 163
I20240503 08:12:10.552060 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.552211 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 163 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.552404 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 163 candidate_status { last_received { term: 162 index: 14310 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.552404 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 163 candidate_status { last_received { term: 162 index: 14310 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.552654 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 163 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.552760 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Leader election lost for term 163. Reason: could not achieve majority
I20240503 08:12:10.557350 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 162. Current term is 163. Ops: []
I20240503 08:12:10.557606 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 162.14310, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.562845 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 162 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:10.562918 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 162 FOLLOWER]: Advancing to term 163
I20240503 08:12:10.563616 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.563817 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 163 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.563989 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 163 candidate_status { last_received { term: 162 index: 14310 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.563997 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 163 candidate_status { last_received { term: 162 index: 14310 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.564004 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 163 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 162. Current term is 163. Ops: []
I20240503 08:12:10.564095 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 163: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:10.564307 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 163 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:10.564420 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 163 FOLLOWER]: Leader election lost for term 163. Reason: could not achieve majority
I20240503 08:12:10.564458 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 162.14310, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.579705 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 162. Current term is 163. Ops: []
I20240503 08:12:10.580008 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 162.14310, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.587481 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.587579 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 163 FOLLOWER]: Advancing to term 164
I20240503 08:12:10.588294 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.588459 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 164 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.588683 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 164 candidate_status { last_received { term: 162 index: 14310 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.588704 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 164 candidate_status { last_received { term: 162 index: 14310 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.588768 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 163 FOLLOWER]: Advancing to term 164
I20240503 08:12:10.589344 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 164 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 164.
I20240503 08:12:10.589571 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 164 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:10.589665 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 FOLLOWER]: Leader election won for term 164
I20240503 08:12:10.589722 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:10.589764 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14310, Committed index: 14310, Last appended: 162.14310, Last appended by leader: 14310, Current term: 164, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.590229 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 162 to 164, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 164 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:10.591724 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 164 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 162. Current term is 164. Ops: []
I20240503 08:12:10.592000 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 162.14310, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.612358 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 162. Current term is 164. Ops: []
I20240503 08:12:10.612650 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 162.14310, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.612742 26516 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 164.14311 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14310
I20240503 08:12:10.620702 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 164 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 162 index: 14310. Preceding OpId from leader: term: 164 index: 14311. (index mismatch)
I20240503 08:12:10.620997 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.622079 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 164 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 162. Current term is 164. Ops: []
I20240503 08:12:10.622291 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 162.14310, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.625509 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 162 LEADER]: Stepping down as leader of term 162
I20240503 08:12:10.625552 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 162 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:10.625597 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14310, Committed index: 14310, Last appended: 162.14310, Last appended by leader: 14310, Current term: 162, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.625706 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 162 FOLLOWER]: Advancing to term 164
I20240503 08:12:10.626365 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 164 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 162 index: 14310. Preceding OpId from leader: term: 164 index: 14311. (index mismatch)
I20240503 08:12:10.626583 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14311, Last known committed idx: 14310, Time since last communication: 0.000s
I20240503 08:12:10.657775 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 164 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:10.657891 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 164 FOLLOWER]: Advancing to term 165
I20240503 08:12:10.658763 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 165 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.658936 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 165 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.659092 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 165 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.659154 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 165 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.659415 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 165 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.659525 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 165 FOLLOWER]: Leader election lost for term 165. Reason: could not achieve majority
I20240503 08:12:10.663790 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 165 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 164. Current term is 165. Ops: []
I20240503 08:12:10.664032 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 164.14311, Next index: 14312, Last known committed idx: 14311, Time since last communication: 0.000s
I20240503 08:12:10.691753 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 165 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.691864 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 165 FOLLOWER]: Advancing to term 166
I20240503 08:12:10.692569 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 166 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.692705 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 166 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 164. Current term is 166. Ops: []
I20240503 08:12:10.692835 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 166 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.692940 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 166 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.692993 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 166 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.692991 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 164.14311, Next index: 14312, Last known committed idx: 14311, Time since last communication: 0.000s
I20240503 08:12:10.693182 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 166 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.693317 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 166 FOLLOWER]: Leader election lost for term 166. Reason: could not achieve majority
I20240503 08:12:10.704560 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 LEADER]: Stepping down as leader of term 164
I20240503 08:12:10.704638 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:10.704689 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14311, Committed index: 14311, Last appended: 164.14311, Last appended by leader: 14311, Current term: 164, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.704804 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 164 FOLLOWER]: Advancing to term 165
I20240503 08:12:10.724088 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 166 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.724192 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 166 FOLLOWER]: Advancing to term 167
I20240503 08:12:10.724850 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 167 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.725020 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 167 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.725289 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 167 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.725335 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 167 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.725375 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 165 FOLLOWER]: Advancing to term 167
I20240503 08:12:10.726008 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 167.
I20240503 08:12:10.726187 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 167 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.726295 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 167 FOLLOWER]: Leader election won for term 167
I20240503 08:12:10.726370 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 167 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:10.726459 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14311, Committed index: 14311, Last appended: 164.14311, Last appended by leader: 14311, Current term: 167, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.726997 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 164 to 167, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 167 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.734019 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 164 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:10.734102 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 164 FOLLOWER]: Advancing to term 165
I20240503 08:12:10.734834 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 165 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.734990 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 165 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.735183 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 165 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.735196 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 165 candidate_status { last_received { term: 164 index: 14311 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.735292 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 165. Current term is 167.
I20240503 08:12:10.735466 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 165 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 165. Current term is 167.
I20240503 08:12:10.735512 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 165 election: Cancelling election due to peer responding with higher term
I20240503 08:12:10.735579 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 165 FOLLOWER]: Advancing to term 167
I20240503 08:12:10.736141 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 167 FOLLOWER]: Leader election lost for term 165. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 165. Current term is 167.
W20240503 08:12:10.749526 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 164 current term: 167 is higher.
W20240503 08:12:10.752781 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 164 current term: 167 is higher.
I20240503 08:12:10.756569 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 167 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 164 index: 14311. Preceding OpId from leader: term: 167 index: 14312. (index mismatch)
I20240503 08:12:10.756870 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14312, Last known committed idx: 14311, Time since last communication: 0.000s
I20240503 08:12:10.760437 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 164 index: 14311. Preceding OpId from leader: term: 167 index: 14312. (index mismatch)
I20240503 08:12:10.760687 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14312, Last known committed idx: 14311, Time since last communication: 0.000s
W20240503 08:12:10.786062 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 164 current term: 167 is higher.
I20240503 08:12:10.842867 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:10.842986 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 167 FOLLOWER]: Advancing to term 168
I20240503 08:12:10.843801 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 168 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.843995 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 168 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.844136 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 168 candidate_status { last_received { term: 167 index: 14312 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.844239 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 168 candidate_status { last_received { term: 167 index: 14312 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.844471 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 168 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.844576 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 168 FOLLOWER]: Leader election lost for term 168. Reason: could not achieve majority
I20240503 08:12:10.846531 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 167 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:10.846592 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 167 FOLLOWER]: Advancing to term 168
I20240503 08:12:10.846968 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 168 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 167. Current term is 168. Ops: []
I20240503 08:12:10.847193 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 167.14312, Next index: 14313, Last known committed idx: 14312, Time since last communication: 0.000s
I20240503 08:12:10.847481 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 168 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.847608 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 168 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.847750 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 168 candidate_status { last_received { term: 167 index: 14312 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.847750 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 168 candidate_status { last_received { term: 167 index: 14312 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.847854 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 168 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 168: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:10.848044 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 168 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:10.848162 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 168 FOLLOWER]: Leader election lost for term 168. Reason: could not achieve majority
I20240503 08:12:10.852617 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 168 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 167. Current term is 168. Ops: []
I20240503 08:12:10.852847 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 167.14312, Next index: 14313, Last known committed idx: 14312, Time since last communication: 0.000s
W20240503 08:12:10.855051 26351 lock_manager.cc:494] Txn 6 has not acquired the partition lock after 1000ms
I20240503 08:12:10.874802 26516 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 167 LEADER]: Stepping down as leader of term 167
I20240503 08:12:10.874887 26516 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 167 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:10.874961 26516 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14312, Committed index: 14312, Last appended: 167.14312, Last appended by leader: 14312, Current term: 167, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.875077 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 167 FOLLOWER]: Advancing to term 168
I20240503 08:12:10.876713 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 168 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:10.876785 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 168 FOLLOWER]: Advancing to term 169
I20240503 08:12:10.877359 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 169 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.877509 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 169 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:10.877708 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 169 candidate_status { last_received { term: 167 index: 14312 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.877707 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 169 candidate_status { last_received { term: 167 index: 14312 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.877818 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 168 FOLLOWER]: Advancing to term 169
I20240503 08:12:10.877832 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 168 FOLLOWER]: Advancing to term 169
I20240503 08:12:10.878316 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 169 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 169.
I20240503 08:12:10.878399 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 169 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 169.
I20240503 08:12:10.878535 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 169 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:10.878650 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 169 FOLLOWER]: Leader election won for term 169
I20240503 08:12:10.878717 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 169 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:10.878774 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14312, Committed index: 14312, Last appended: 167.14312, Last appended by leader: 14312, Current term: 169, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.879258 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 167 to 169, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 169 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:10.906714 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 169 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 167 index: 14312. Preceding OpId from leader: term: 169 index: 14313. (index mismatch)
I20240503 08:12:10.907009 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14313, Last known committed idx: 14312, Time since last communication: 0.000s
I20240503 08:12:10.915191 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 169 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 167 index: 14312. Preceding OpId from leader: term: 169 index: 14313. (index mismatch)
I20240503 08:12:10.915490 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14313, Last known committed idx: 14312, Time since last communication: 0.000s
I20240503 08:12:10.938792 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 169 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:10.938896 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 169 FOLLOWER]: Advancing to term 170
I20240503 08:12:10.939673 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.939824 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 170 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.939947 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 170 candidate_status { last_received { term: 169 index: 14313 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.940007 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 170 candidate_status { last_received { term: 169 index: 14313 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:10.940217 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 170 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:10.940322 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Leader election lost for term 170. Reason: could not achieve majority
I20240503 08:12:10.942152 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 169. Current term is 170. Ops: []
I20240503 08:12:10.942487 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 169.14313, Next index: 14314, Last known committed idx: 14313, Time since last communication: 0.000s
I20240503 08:12:10.942673 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 169 LEADER]: Stepping down as leader of term 169
I20240503 08:12:10.942713 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 169 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:10.942771 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14313, Committed index: 14313, Last appended: 169.14313, Last appended by leader: 14313, Current term: 169, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.942847 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 169 FOLLOWER]: Advancing to term 170
I20240503 08:12:10.946530 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 169 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:10.946640 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 169 FOLLOWER]: Advancing to term 170
I20240503 08:12:10.947326 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 170 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.947475 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 170 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:10.947748 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 170 candidate_status { last_received { term: 169 index: 14313 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:10.947726 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 170 candidate_status { last_received { term: 169 index: 14313 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:10.947824 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 170: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:10.948437 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 170 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 170.
I20240503 08:12:10.948693 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 170 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:10.948869 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 170 FOLLOWER]: Leader election won for term 170
I20240503 08:12:10.948941 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 170 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:10.949074 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14313, Committed index: 14313, Last appended: 169.14313, Last appended by leader: 14313, Current term: 170, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:10.949657 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 169 to 170, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 170 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:10.976163 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 169 index: 14313. Preceding OpId from leader: term: 170 index: 14314. (index mismatch)
I20240503 08:12:10.976445 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14314, Last known committed idx: 14313, Time since last communication: 0.000s
I20240503 08:12:10.982259 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 170 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 169 index: 14313. Preceding OpId from leader: term: 170 index: 14314. (index mismatch)
I20240503 08:12:10.982506 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14314, Last known committed idx: 14313, Time since last communication: 0.000s
I20240503 08:12:11.008339 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:11.008435 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 170 FOLLOWER]: Advancing to term 171
I20240503 08:12:11.009104 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.009357 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 171 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.009457 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 171 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.009457 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 171 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.009716 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 171 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.009825 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Leader election lost for term 171. Reason: could not achieve majority
I20240503 08:12:11.014231 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 170. Current term is 171. Ops: []
I20240503 08:12:11.014534 26563 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.036154 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 170 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:11.036255 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 170 FOLLOWER]: Advancing to term 171
I20240503 08:12:11.036931 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 171 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.037101 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 171 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.037328 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 171 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.037334 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 171 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.037449 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 171: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:11.037631 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 171 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.037732 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 171 FOLLOWER]: Leader election lost for term 171. Reason: could not achieve majority
I20240503 08:12:11.038822 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 171 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 170. Current term is 171. Ops: []
I20240503 08:12:11.039072 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.043162 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 170. Current term is 171. Ops: []
I20240503 08:12:11.043541 26563 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.050668 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.050766 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 171 FOLLOWER]: Advancing to term 172
I20240503 08:12:11.051450 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.051615 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 172 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.051846 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 172 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.051851 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 172 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.051983 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 171 FOLLOWER]: Advancing to term 172
I20240503 08:12:11.052582 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 172 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 172.
I20240503 08:12:11.052767 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 172 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.052865 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 FOLLOWER]: Leader election won for term 172
I20240503 08:12:11.052937 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.052980 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14314, Committed index: 14314, Last appended: 170.14314, Last appended by leader: 14314, Current term: 172, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.053498 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 170 to 172, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 172 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:11.070808 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 172 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 170. Current term is 172. Ops: []
I20240503 08:12:11.071085 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.077137 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 170. Current term is 172. Ops: []
I20240503 08:12:11.077415 26563 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.077512 26563 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 172.14315 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14314
I20240503 08:12:11.085654 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 172 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.085747 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 172 FOLLOWER]: Advancing to term 173
I20240503 08:12:11.086400 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.086580 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 173 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.086795 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 173 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.086850 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 173 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.086803 26284 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 170 LEADER]: Stepping down as leader of term 170
I20240503 08:12:11.086956 26284 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 170 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:11.087003 26284 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14314, Committed index: 14314, Last appended: 170.14314, Last appended by leader: 14314, Current term: 170, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.087071 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 173 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.087082 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 170 FOLLOWER]: Advancing to term 172
I20240503 08:12:11.087213 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Leader election lost for term 173. Reason: could not achieve majority
I20240503 08:12:11.087728 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 172 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 170 index: 14314. Preceding OpId from leader: term: 172 index: 14315. (index mismatch)
I20240503 08:12:11.087971 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.089926 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 172. Current term is 173. Ops: []
I20240503 08:12:11.090173 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.110148 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 172. Current term is 173. Ops: [172.14315-172.14315]
I20240503 08:12:11.110419 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.134405 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 172. Current term is 173. Ops: [172.14315-172.14315]
I20240503 08:12:11.134781 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.143159 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.143275 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 173 FOLLOWER]: Advancing to term 174
I20240503 08:12:11.143993 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.144192 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 174 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.144441 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 174 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.144472 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 174 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.144687 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 174 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.144781 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Leader election lost for term 174. Reason: could not achieve majority
I20240503 08:12:11.169242 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 172. Current term is 174. Ops: [172.14315-172.14315]
I20240503 08:12:11.169533 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 170.14314, Next index: 14315, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.181210 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 LEADER]: Stepping down as leader of term 172
I20240503 08:12:11.181330 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.181408 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14315, Committed index: 14315, Last appended: 172.14315, Last appended by leader: 14315, Current term: 172, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.181532 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 172 FOLLOWER]: Advancing to term 173
I20240503 08:12:11.194018 26563 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 172 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:11.194126 26563 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 172 FOLLOWER]: Advancing to term 173
I20240503 08:12:11.194944 26563 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 173 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.195099 26563 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 173 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.195326 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 173 candidate_status { last_received { term: 172 index: 14315 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.195326 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 173 candidate_status { last_received { term: 172 index: 14315 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.195430 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 173. Current term is 174.
I20240503 08:12:11.195590 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 173 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 173. Current term is 174.
I20240503 08:12:11.195643 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 173 election: Cancelling election due to peer responding with higher term
I20240503 08:12:11.195722 26563 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 173 FOLLOWER]: Advancing to term 174
I20240503 08:12:11.195998 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 173 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 173.
I20240503 08:12:11.196297 26563 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 174 FOLLOWER]: Leader election lost for term 173. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 173. Current term is 174.
I20240503 08:12:11.227228 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 173 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.227339 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 173 FOLLOWER]: Advancing to term 174
I20240503 08:12:11.228027 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 174 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.228201 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 174 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.228472 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 174 candidate_status { last_received { term: 172 index: 14315 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.228509 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 174 candidate_status { last_received { term: 172 index: 14315 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.228624 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 174: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:11.229162 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 174 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 174.
I20240503 08:12:11.229349 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 174 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:11.229445 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 174 FOLLOWER]: Leader election won for term 174
I20240503 08:12:11.229496 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 174 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.229535 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14315, Committed index: 14315, Last appended: 172.14315, Last appended by leader: 14315, Current term: 174, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.229977 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 172 to 174. New cstate: current_term: 174 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:11.238446 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 173 current term: 174 is higher.
I20240503 08:12:11.242357 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.242475 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 174 FOLLOWER]: Advancing to term 175
I20240503 08:12:11.243204 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 175 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.243386 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 175 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.243631 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 175 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.243631 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 175 candidate_status { last_received { term: 170 index: 14314 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.243788 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 174 FOLLOWER]: Advancing to term 175
I20240503 08:12:11.244585 26284 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 175 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 175 because replica has last-logged OpId of term: 172 index: 14315, which is greater than that of the candidate, which has last-logged OpId of term: 170 index: 14314.
I20240503 08:12:11.244939 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 175 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.245095 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 175 FOLLOWER]: Leader election lost for term 175. Reason: could not achieve majority
W20240503 08:12:11.245673 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 174 current term: 174 is higher.
I20240503 08:12:11.257480 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 175 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 174. Current term is 175. Ops: []
I20240503 08:12:11.257764 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14316, Last known committed idx: 14315, Time since last communication: 0.000s
I20240503 08:12:11.259379 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 175 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 174. Current term is 175. Ops: []
I20240503 08:12:11.259624 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14316, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.274328 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 175 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.274430 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 175 FOLLOWER]: Advancing to term 176
I20240503 08:12:11.275125 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 176 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.275297 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 176 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.275568 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 176 candidate_status { last_received { term: 172 index: 14315 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.275604 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 176 candidate_status { last_received { term: 172 index: 14315 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.275661 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 175 FOLLOWER]: Advancing to term 176
I20240503 08:12:11.276295 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 176 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 176.
I20240503 08:12:11.276531 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 176 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.276646 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 176 FOLLOWER]: Leader election won for term 176
I20240503 08:12:11.276723 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 176 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:11.276782 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14315, Committed index: 14315, Last appended: 172.14315, Last appended by leader: 14315, Current term: 176, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.277242 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 174 to 176, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 176 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:11.286031 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 174 LEADER]: Stepping down as leader of term 174
I20240503 08:12:11.286115 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 174 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.286195 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14315, Committed index: 14315, Last appended: 174.14316, Last appended by leader: 14316, Current term: 174, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.286324 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 174 FOLLOWER]: Advancing to term 175
I20240503 08:12:11.309273 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 175 FOLLOWER]: Advancing to term 176
I20240503 08:12:11.310007 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 176 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 174 index: 14316. Preceding OpId from leader: term: 176 index: 14316. (term mismatch)
I20240503 08:12:11.310052 26133 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14315
I20240503 08:12:11.310065 26133 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 174.14316
I20240503 08:12:11.310081 26133 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 176 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:11.310372 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14316, Last known committed idx: 14315, Time since last communication: 0.000s
I20240503 08:12:11.313886 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 176 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 170 index: 14314. Preceding OpId from leader: term: 176 index: 14316. (index mismatch)
I20240503 08:12:11.314152 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14316, Last known committed idx: 14314, Time since last communication: 0.000s
I20240503 08:12:11.402554 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 176 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:11.402683 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 176 FOLLOWER]: Advancing to term 177
I20240503 08:12:11.403417 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 177 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.403541 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 177 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 176. Current term is 177. Ops: []
I20240503 08:12:11.403656 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 177 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.403738 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 177 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.403779 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 177 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.403870 26563 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 176.14316, Next index: 14317, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.403924 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 177 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.404039 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 177 FOLLOWER]: Leader election lost for term 177. Reason: could not achieve majority
I20240503 08:12:11.433880 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 177 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 176. Current term is 177. Ops: []
I20240503 08:12:11.434180 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 176.14316, Next index: 14317, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.435894 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 176 LEADER]: Stepping down as leader of term 176
I20240503 08:12:11.435966 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 176 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:11.436033 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14316, Committed index: 14316, Last appended: 176.14316, Last appended by leader: 14316, Current term: 176, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.436159 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 176 FOLLOWER]: Advancing to term 177
I20240503 08:12:11.446251 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 177 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.446382 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 177 FOLLOWER]: Advancing to term 178
I20240503 08:12:11.447153 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 178 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.447340 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 178 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.447552 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 178 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.447552 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 178 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.447672 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 177 FOLLOWER]: Advancing to term 178
I20240503 08:12:11.448239 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 178 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 178.
I20240503 08:12:11.448439 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 178 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:11.448531 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 178 FOLLOWER]: Leader election won for term 178
I20240503 08:12:11.448581 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 178 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.448642 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14316, Committed index: 14316, Last appended: 176.14316, Last appended by leader: 14316, Current term: 178, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.449134 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 176 to 178, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 178 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:11.450873 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 176 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:11.450938 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 176 FOLLOWER]: Advancing to term 177
I20240503 08:12:11.451510 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 177 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.451637 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 177 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.451838 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 177 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.451903 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 177 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.451982 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 178 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 177. Current term is 178.
I20240503 08:12:11.452167 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 177 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 178 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 177. Current term is 178.
I20240503 08:12:11.452212 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 177 election: Cancelling election due to peer responding with higher term
I20240503 08:12:11.452273 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 177 FOLLOWER]: Advancing to term 178
I20240503 08:12:11.452767 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 178 FOLLOWER]: Leader election lost for term 177. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 178 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 177. Current term is 178.
I20240503 08:12:11.482159 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 178 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 176 index: 14316. Preceding OpId from leader: term: 178 index: 14317. (index mismatch)
I20240503 08:12:11.482601 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14317, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.483841 26563 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 178 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.483919 26563 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 178 FOLLOWER]: Advancing to term 179
I20240503 08:12:11.484645 26563 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.484828 26563 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 179 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.484845 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 178. Current term is 179. Ops: []
I20240503 08:12:11.484913 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 179 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.485002 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 179 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.485100 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14317, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.485170 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 179 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.485304 26563 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Leader election lost for term 179. Reason: could not achieve majority
I20240503 08:12:11.487947 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 178. Current term is 179. Ops: [178.14317-178.14317]
I20240503 08:12:11.488154 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 176.14316, Next index: 14317, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.510042 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 178 LEADER]: Stepping down as leader of term 178
I20240503 08:12:11.510133 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 178 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.510205 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14317, Committed index: 14317, Last appended: 178.14317, Last appended by leader: 14317, Current term: 178, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.510344 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 178 FOLLOWER]: Advancing to term 179
I20240503 08:12:11.514816 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 178 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:11.514914 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 178 FOLLOWER]: Advancing to term 179
I20240503 08:12:11.515542 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 179 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.515699 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 179 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.515920 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 179 candidate_status { last_received { term: 178 index: 14317 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.515937 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 179 candidate_status { last_received { term: 178 index: 14317 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.516008 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 179: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:11.516533 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 179 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 179.
I20240503 08:12:11.516734 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 179 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.516832 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 179 FOLLOWER]: Leader election won for term 179
I20240503 08:12:11.516896 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 179 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.516953 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14317, Committed index: 14317, Last appended: 178.14317, Last appended by leader: 14317, Current term: 179, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.517531 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 178 to 179, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 179 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:11.526778 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.526856 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 179 FOLLOWER]: Advancing to term 180
I20240503 08:12:11.527487 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 180 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.527635 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 180 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.527918 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 180 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.527918 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 180 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.528033 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 179 FOLLOWER]: Advancing to term 180
I20240503 08:12:11.528591 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 180 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for term 180 because replica has last-logged OpId of term: 178 index: 14317, which is greater than that of the candidate, which has last-logged OpId of term: 176 index: 14316.
I20240503 08:12:11.528795 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 180 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.528885 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 180 FOLLOWER]: Leader election lost for term 180. Reason: could not achieve majority
I20240503 08:12:11.541949 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 180 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 179. Current term is 180. Ops: []
I20240503 08:12:11.542215 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14318, Last known committed idx: 14317, Time since last communication: 0.000s
I20240503 08:12:11.548843 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 180 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 179. Current term is 180. Ops: []
I20240503 08:12:11.549111 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14318, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.561025 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 180 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.561129 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 180 FOLLOWER]: Advancing to term 181
I20240503 08:12:11.561766 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.561924 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 181 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.562140 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 181 candidate_status { last_received { term: 178 index: 14317 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.562201 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 180 FOLLOWER]: Advancing to term 181
I20240503 08:12:11.562155 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 181 candidate_status { last_received { term: 178 index: 14317 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.562752 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 181 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 181.
I20240503 08:12:11.562954 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 181 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.563055 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 FOLLOWER]: Leader election won for term 181
I20240503 08:12:11.563102 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.563162 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14317, Committed index: 14317, Last appended: 178.14317, Last appended by leader: 14317, Current term: 181, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.563645 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 179 to 181, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 181 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:11.575325 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 179. Current term is 181. Ops: [179.14318-179.14318]
I20240503 08:12:11.575667 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 178.14317, Next index: 14318, Last known committed idx: 14317, Time since last communication: 0.000s
I20240503 08:12:11.575789 26516 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 181.14318 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14317
I20240503 08:12:11.576074 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 181 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 179. Current term is 181. Ops: [178.14317-179.14318]
I20240503 08:12:11.576297 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 176.14316, Next index: 14317, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.579432 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 179 LEADER]: Stepping down as leader of term 179
I20240503 08:12:11.579490 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 179 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.579545 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14317, Committed index: 14317, Last appended: 179.14318, Last appended by leader: 14318, Current term: 179, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.579631 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 179 FOLLOWER]: Advancing to term 180
I20240503 08:12:11.593858 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 181 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.593961 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 181 FOLLOWER]: Advancing to term 182
I20240503 08:12:11.594659 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 182 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.594828 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 182 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.595036 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 182 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.595043 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 182 candidate_status { last_received { term: 176 index: 14316 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.595145 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 180 FOLLOWER]: Advancing to term 182
I20240503 08:12:11.595744 26133 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 182 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for term 182 because replica has last-logged OpId of term: 179 index: 14318, which is greater than that of the candidate, which has last-logged OpId of term: 176 index: 14316.
I20240503 08:12:11.595955 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 182 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.596063 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 182 FOLLOWER]: Leader election lost for term 182. Reason: could not achieve majority
I20240503 08:12:11.596649 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 182 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 181. Current term is 182. Ops: []
I20240503 08:12:11.596886 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14318, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.599062 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 182 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 181. Current term is 182. Ops: []
I20240503 08:12:11.599303 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14318, Last known committed idx: 14317, Time since last communication: 0.000s
I20240503 08:12:11.599391 26334 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 179.14318 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14317
I20240503 08:12:11.602859 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 182 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.602924 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 182 FOLLOWER]: Advancing to term 183
I20240503 08:12:11.603514 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 183 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.603641 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 183 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.603827 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 183 candidate_status { last_received { term: 179 index: 14318 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.603906 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 182 FOLLOWER]: Advancing to term 183
I20240503 08:12:11.603835 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 183 candidate_status { last_received { term: 179 index: 14318 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.604486 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 183 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 183.
I20240503 08:12:11.604665 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 183 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:11.604749 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 183 FOLLOWER]: Leader election won for term 183
I20240503 08:12:11.604902 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 183 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.604975 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14317, Committed index: 14317, Last appended: 179.14318, Last appended by leader: 14318, Current term: 183, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.605505 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 181 to 183, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 183 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:11.606096 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 LEADER]: Stepping down as leader of term 181
I20240503 08:12:11.606140 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.606200 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14317, Committed index: 14317, Last appended: 181.14318, Last appended by leader: 14318, Current term: 181, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.606298 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 181 FOLLOWER]: Advancing to term 182
W20240503 08:12:11.618480 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 181 current term: 183 is higher.
W20240503 08:12:11.620659 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 181 current term: 183 is higher.
I20240503 08:12:11.632169 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 182 FOLLOWER]: Advancing to term 183
I20240503 08:12:11.632887 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 183 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 181 index: 14318. Preceding OpId from leader: term: 183 index: 14319. (index mismatch)
I20240503 08:12:11.633170 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14319, Last known committed idx: 14317, Time since last communication: 0.000s
I20240503 08:12:11.633261 26515 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 181.14318 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14317
I20240503 08:12:11.633445 26210 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting all ops after (but not including) 14317
I20240503 08:12:11.633483 26210 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting uncommitted NO_OP operation due to leader change: 181.14318
I20240503 08:12:11.633500 26210 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 183 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:11.641786 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 183 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 176 index: 14316. Preceding OpId from leader: term: 183 index: 14319. (index mismatch)
I20240503 08:12:11.642033 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14319, Last known committed idx: 14316, Time since last communication: 0.000s
I20240503 08:12:11.665040 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 183 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:11.665148 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 183 FOLLOWER]: Advancing to term 184
I20240503 08:12:11.665803 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.665969 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 184 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.666194 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 184 candidate_status { last_received { term: 183 index: 14319 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.666194 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 184 candidate_status { last_received { term: 183 index: 14319 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.666481 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 184 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.666597 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Leader election lost for term 184. Reason: could not achieve majority
I20240503 08:12:11.670440 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 183. Current term is 184. Ops: []
I20240503 08:12:11.670702 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 183.14319, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.673063 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 183 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:11.673116 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 183 FOLLOWER]: Advancing to term 184
I20240503 08:12:11.673750 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 184 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.673882 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 184 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.674073 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 184 candidate_status { last_received { term: 183 index: 14319 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.674080 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 184 candidate_status { last_received { term: 183 index: 14319 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.674186 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 184: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:11.674341 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 184 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.674437 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 184 FOLLOWER]: Leader election lost for term 184. Reason: could not achieve majority
I20240503 08:12:11.676743 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 184 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 183. Current term is 184. Ops: []
I20240503 08:12:11.676991 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 183.14319, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.694887 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 183. Current term is 184. Ops: []
I20240503 08:12:11.695190 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 183.14319, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.702715 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 184 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 183. Current term is 184. Ops: []
I20240503 08:12:11.703003 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 183.14319, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.703112 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.703173 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 184 FOLLOWER]: Advancing to term 185
I20240503 08:12:11.703819 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.703974 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 185 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.704162 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 185 candidate_status { last_received { term: 183 index: 14319 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.704171 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 185 candidate_status { last_received { term: 183 index: 14319 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.704250 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 184 FOLLOWER]: Advancing to term 185
I20240503 08:12:11.704921 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 185 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 185.
I20240503 08:12:11.705138 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 185 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.705236 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 FOLLOWER]: Leader election won for term 185
I20240503 08:12:11.705299 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.705448 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14319, Committed index: 14319, Last appended: 183.14319, Last appended by leader: 14319, Current term: 185, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.705916 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 183 to 185, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 185 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:11.723510 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 183. Current term is 185. Ops: []
I20240503 08:12:11.723791 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 183.14319, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.723912 26516 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 185.14320 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14319
I20240503 08:12:11.735584 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 185 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 183. Current term is 185. Ops: []
I20240503 08:12:11.735759 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 185 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 183 index: 14319. Preceding OpId from leader: term: 185 index: 14320. (index mismatch)
I20240503 08:12:11.735863 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 183.14319, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.735944 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.738196 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 183 LEADER]: Stepping down as leader of term 183
I20240503 08:12:11.738255 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 183 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.738301 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14319, Committed index: 14319, Last appended: 183.14319, Last appended by leader: 14319, Current term: 183, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.738402 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 183 FOLLOWER]: Advancing to term 185
I20240503 08:12:11.739064 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 185 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 183 index: 14319. Preceding OpId from leader: term: 185 index: 14320. (index mismatch)
I20240503 08:12:11.739342 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14320, Last known committed idx: 14319, Time since last communication: 0.000s
I20240503 08:12:11.767788 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 185 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:11.767913 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 185 FOLLOWER]: Advancing to term 186
I20240503 08:12:11.768604 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 186 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.768748 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 186 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 185. Current term is 186. Ops: []
I20240503 08:12:11.768779 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 186 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.768918 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 186 candidate_status { last_received { term: 185 index: 14320 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.768992 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 185.14320, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.768996 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 186 candidate_status { last_received { term: 185 index: 14320 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.769220 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 186 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.769308 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 186 FOLLOWER]: Leader election lost for term 186. Reason: could not achieve majority
I20240503 08:12:11.770797 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 185 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:11.770846 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 185 FOLLOWER]: Advancing to term 186
I20240503 08:12:11.771407 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 186 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.771522 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 186 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.771687 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 186 candidate_status { last_received { term: 185 index: 14320 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.771698 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 186 candidate_status { last_received { term: 185 index: 14320 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.771790 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 186 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 186: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:11.771912 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 186 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.772029 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 186 FOLLOWER]: Leader election lost for term 186. Reason: could not achieve majority
I20240503 08:12:11.773705 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 186 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 185. Current term is 186. Ops: []
I20240503 08:12:11.773901 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 185.14320, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.795691 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 186 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 185. Current term is 186. Ops: []
I20240503 08:12:11.795980 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 185.14320, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.800627 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 186 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 185. Current term is 186. Ops: []
I20240503 08:12:11.800963 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 185.14320, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.804036 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 186 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.804105 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 186 FOLLOWER]: Advancing to term 187
I20240503 08:12:11.804749 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.804880 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 187 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.805096 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 187 candidate_status { last_received { term: 185 index: 14320 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.805157 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 186 FOLLOWER]: Advancing to term 187
I20240503 08:12:11.805105 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 187 candidate_status { last_received { term: 185 index: 14320 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.805753 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 187 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 187.
I20240503 08:12:11.805928 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 187 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:11.806022 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 FOLLOWER]: Leader election won for term 187
I20240503 08:12:11.806082 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.806140 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14320, Committed index: 14320, Last appended: 185.14320, Last appended by leader: 14320, Current term: 187, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.806587 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 185 to 187, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 187 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:11.811827 26502 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 184 current term: 187 is higher.
I20240503 08:12:11.820044 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 187 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 185. Current term is 187. Ops: []
I20240503 08:12:11.820333 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 185.14320, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.825664 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 185. Current term is 187. Ops: []
I20240503 08:12:11.825944 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 185.14320, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.826043 26556 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 187.14321 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14320
I20240503 08:12:11.836148 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 187 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 185 index: 14320. Preceding OpId from leader: term: 187 index: 14321. (index mismatch)
I20240503 08:12:11.836428 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
I20240503 08:12:11.842305 26210 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 LEADER]: Stepping down as leader of term 185
I20240503 08:12:11.842381 26210 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.842429 26210 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14320, Committed index: 14320, Last appended: 185.14320, Last appended by leader: 14320, Current term: 185, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.842540 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 185 FOLLOWER]: Advancing to term 187
I20240503 08:12:11.843183 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 187 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 185 index: 14320. Preceding OpId from leader: term: 187 index: 14321. (index mismatch)
I20240503 08:12:11.843433 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14321, Last known committed idx: 14320, Time since last communication: 0.000s
W20240503 08:12:11.855190 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 184 current term: 187 is higher.
W20240503 08:12:11.855870 26349 lock_manager.cc:494] Txn 6 has not acquired the partition lock after 1000ms
I20240503 08:12:11.873860 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 187 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:11.873952 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 187 FOLLOWER]: Advancing to term 188
I20240503 08:12:11.874663 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.874789 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 187. Current term is 188. Ops: []
I20240503 08:12:11.874840 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 188 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.875010 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 187.14321, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20240503 08:12:11.875005 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 188 candidate_status { last_received { term: 187 index: 14321 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.875006 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 188 candidate_status { last_received { term: 187 index: 14321 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.875303 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 188 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.875391 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Leader election lost for term 188. Reason: could not achieve majority
W20240503 08:12:11.882416 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 185 current term: 187 is higher.
I20240503 08:12:11.893275 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 187 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:11.893363 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 187 FOLLOWER]: Advancing to term 188
I20240503 08:12:11.893992 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 188 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.894138 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 188 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.894356 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 188 candidate_status { last_received { term: 187 index: 14321 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.894363 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 188 candidate_status { last_received { term: 187 index: 14321 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.894477 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 188: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:11.894635 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 188 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.894730 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 188 FOLLOWER]: Leader election lost for term 188. Reason: could not achieve majority
I20240503 08:12:11.897693 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 188 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 187. Current term is 188. Ops: []
I20240503 08:12:11.897920 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 187.14321, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20240503 08:12:11.906030 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 187. Current term is 188. Ops: []
I20240503 08:12:11.906302 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 187.14321, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20240503 08:12:11.919514 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:11.919613 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 188 FOLLOWER]: Advancing to term 189
I20240503 08:12:11.920284 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.920434 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 189 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:11.920637 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 189 candidate_status { last_received { term: 187 index: 14321 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.920650 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 189 candidate_status { last_received { term: 187 index: 14321 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.920753 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 188 FOLLOWER]: Advancing to term 189
I20240503 08:12:11.921355 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 189 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 189.
I20240503 08:12:11.921553 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 189 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.921643 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 FOLLOWER]: Leader election won for term 189
I20240503 08:12:11.921722 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:11.921785 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14321, Committed index: 14321, Last appended: 187.14321, Last appended by leader: 14321, Current term: 189, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.922242 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 187 to 189, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 189 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:11.923584 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 185 current term: 187 is higher.
I20240503 08:12:11.926929 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 189 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 187. Current term is 189. Ops: []
I20240503 08:12:11.927156 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 187.14321, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20240503 08:12:11.927999 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 187. Current term is 189. Ops: []
I20240503 08:12:11.928270 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 187.14321, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20240503 08:12:11.928364 26516 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 189.14322 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14321
I20240503 08:12:11.947934 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 LEADER]: Stepping down as leader of term 187
I20240503 08:12:11.948019 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:11.948093 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14321, Committed index: 14321, Last appended: 187.14321, Last appended by leader: 14321, Current term: 187, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.948210 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 187 FOLLOWER]: Advancing to term 189
I20240503 08:12:11.948894 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 189 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 187 index: 14321. Preceding OpId from leader: term: 189 index: 14322. (index mismatch)
I20240503 08:12:11.949183 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
W20240503 08:12:11.955843 26334 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 186 current term: 189 is higher.
I20240503 08:12:11.957680 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 189 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 187 index: 14321. Preceding OpId from leader: term: 189 index: 14322. (index mismatch)
I20240503 08:12:11.957906 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20240503 08:12:11.980654 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 189 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:11.980758 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 189 FOLLOWER]: Advancing to term 190
I20240503 08:12:11.981459 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.981614 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 190 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.981827 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 190 candidate_status { last_received { term: 189 index: 14322 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:11.981845 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 190 candidate_status { last_received { term: 189 index: 14322 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.982081 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 190 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:11.982206 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Leader election lost for term 190. Reason: could not achieve majority
I20240503 08:12:11.984278 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 189. Current term is 190. Ops: []
I20240503 08:12:11.984532 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 189.14322, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:11.989166 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 189 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:11.989243 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 189 FOLLOWER]: Advancing to term 190
I20240503 08:12:11.989868 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 190 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:11.990010 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 190 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:11.990223 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 190 candidate_status { last_received { term: 189 index: 14322 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:11.990234 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 190 candidate_status { last_received { term: 189 index: 14322 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:11.990326 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 190: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:11.990517 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 190 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:11.990625 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 190 FOLLOWER]: Leader election lost for term 190. Reason: could not achieve majority
I20240503 08:12:11.993285 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 190 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 189. Current term is 190. Ops: []
I20240503 08:12:11.993515 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 189.14322, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
W20240503 08:12:11.997056 26334 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 187 current term: 189 is higher.
W20240503 08:12:11.998224 26334 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 187 current term: 189 is higher.
I20240503 08:12:12.009248 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 189. Current term is 190. Ops: []
I20240503 08:12:12.009532 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 189.14322, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:12.015539 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 190 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 189. Current term is 190. Ops: []
I20240503 08:12:12.015795 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 189.14322, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:12.019876 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.019934 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 190 FOLLOWER]: Advancing to term 191
I20240503 08:12:12.020558 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.020691 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 191 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.020913 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 191 candidate_status { last_received { term: 189 index: 14322 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.020915 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 191 candidate_status { last_received { term: 189 index: 14322 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.021000 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 190 FOLLOWER]: Advancing to term 191
I20240503 08:12:12.021528 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 191 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 191.
I20240503 08:12:12.021690 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 191 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:12.021788 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 FOLLOWER]: Leader election won for term 191
I20240503 08:12:12.021838 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:12.021874 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14322, Committed index: 14322, Last appended: 189.14322, Last appended by leader: 14322, Current term: 191, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.022282 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 189 to 191, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 191 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.034340 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 189. Current term is 191. Ops: []
I20240503 08:12:12.034644 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 189.14322, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:12.034762 26378 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 191.14323 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14322
I20240503 08:12:12.038933 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 191 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 189. Current term is 191. Ops: []
I20240503 08:12:12.039206 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 189.14322, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:12.043762 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 LEADER]: Stepping down as leader of term 189
I20240503 08:12:12.043853 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.044005 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14322, Committed index: 14322, Last appended: 189.14322, Last appended by leader: 14322, Current term: 189, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.044138 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 189 FOLLOWER]: Advancing to term 190
I20240503 08:12:12.054033 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 191 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 189 index: 14322. Preceding OpId from leader: term: 191 index: 14323. (index mismatch)
I20240503 08:12:12.054461 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:12.056789 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 190 FOLLOWER]: Advancing to term 191
I20240503 08:12:12.057448 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 191 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 189 index: 14322. Preceding OpId from leader: term: 191 index: 14323. (index mismatch)
I20240503 08:12:12.057704 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14323, Last known committed idx: 14322, Time since last communication: 0.000s
I20240503 08:12:12.086041 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 191 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:12.086141 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 191 FOLLOWER]: Advancing to term 192
I20240503 08:12:12.086850 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.086989 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 191. Current term is 192. Ops: []
I20240503 08:12:12.087026 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 192 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.087194 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 192 candidate_status { last_received { term: 191 index: 14323 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.087240 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 191.14323, Next index: 14324, Last known committed idx: 14323, Time since last communication: 0.000s
I20240503 08:12:12.087260 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 192 candidate_status { last_received { term: 191 index: 14323 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.087445 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 192 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:12.087548 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Leader election lost for term 192. Reason: could not achieve majority
I20240503 08:12:12.088256 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 191 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:12.088292 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 191 FOLLOWER]: Advancing to term 192
I20240503 08:12:12.088940 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 192 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.089033 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 192 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 191. Current term is 192. Ops: []
I20240503 08:12:12.089069 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 192 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.089229 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 192 candidate_status { last_received { term: 191 index: 14323 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.089241 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 192 candidate_status { last_received { term: 191 index: 14323 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.089241 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 191.14323, Next index: 14324, Last known committed idx: 14323, Time since last communication: 0.000s
I20240503 08:12:12.089360 26285 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 192: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:12.089522 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 192 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.089615 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 192 FOLLOWER]: Leader election lost for term 192. Reason: could not achieve majority
W20240503 08:12:12.093530 26516 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 188 current term: 191 is higher.
W20240503 08:12:12.119848 26516 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 189 current term: 191 is higher.
I20240503 08:12:12.123380 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 191. Current term is 192. Ops: []
I20240503 08:12:12.123373 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 192 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 191. Current term is 192. Ops: []
I20240503 08:12:12.123705 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 191.14323, Next index: 14324, Last known committed idx: 14323, Time since last communication: 0.000s
I20240503 08:12:12.123811 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 191.14323, Next index: 14324, Last known committed idx: 14323, Time since last communication: 0.000s
I20240503 08:12:12.126031 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.126098 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 192 FOLLOWER]: Advancing to term 193
I20240503 08:12:12.126770 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 193 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.126950 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 193 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.127142 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 193 candidate_status { last_received { term: 191 index: 14323 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.127142 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 193 candidate_status { last_received { term: 191 index: 14323 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.127225 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 192 FOLLOWER]: Advancing to term 193
I20240503 08:12:12.127768 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 193 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 193.
I20240503 08:12:12.128007 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 193 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:12.128101 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 193 FOLLOWER]: Leader election won for term 193
I20240503 08:12:12.128166 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 193 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.128219 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14323, Committed index: 14323, Last appended: 191.14323, Last appended by leader: 14323, Current term: 193, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.128664 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 191 to 193, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 193 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:12.150090 26516 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 189 current term: 191 is higher.
I20240503 08:12:12.152063 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 LEADER]: Stepping down as leader of term 191
I20240503 08:12:12.152122 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:12.152189 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14323, Committed index: 14323, Last appended: 191.14323, Last appended by leader: 14323, Current term: 191, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.152313 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 191 FOLLOWER]: Advancing to term 193
I20240503 08:12:12.152966 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 193 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 191 index: 14323. Preceding OpId from leader: term: 193 index: 14324. (index mismatch)
I20240503 08:12:12.153252 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14324, Last known committed idx: 14323, Time since last communication: 0.000s
I20240503 08:12:12.163170 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 193 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 191 index: 14323. Preceding OpId from leader: term: 193 index: 14324. (index mismatch)
I20240503 08:12:12.163443 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14324, Last known committed idx: 14323, Time since last communication: 0.000s
I20240503 08:12:12.193867 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 193 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:12.193958 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 193 FOLLOWER]: Advancing to term 194
I20240503 08:12:12.194589 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 194 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.194782 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 194 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.194928 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 194 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.194978 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 194 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 193. Current term is 194. Ops: []
I20240503 08:12:12.194955 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 194 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.195181 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 194 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.195277 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 194 FOLLOWER]: Leader election lost for term 194. Reason: could not achieve majority
I20240503 08:12:12.195271 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 193.14324, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.226954 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 194 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 193. Current term is 194. Ops: []
I20240503 08:12:12.227233 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 193.14324, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.238022 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 194 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.238147 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 194 FOLLOWER]: Advancing to term 195
I20240503 08:12:12.238898 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.239054 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 195 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.239313 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 195 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.239488 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 193 FOLLOWER]: Advancing to term 195
I20240503 08:12:12.239322 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 195 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.240391 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 195 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 195.
I20240503 08:12:12.240454 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 195 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.240482 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 195 FOLLOWER]: Advancing to term 196
I20240503 08:12:12.241453 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 195 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.241568 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 FOLLOWER]: Leader election won for term 195
I20240503 08:12:12.241535 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.241636 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.241714 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14324, Committed index: 14324, Last appended: 193.14324, Last appended by leader: 14324, Current term: 195, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.241788 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 196 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.241914 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 196 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.241981 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 193. Current term is 196. Ops: []
I20240503 08:12:12.242020 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 196 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.242166 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 196 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.242278 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Leader election lost for term 196. Reason: could not achieve majority
I20240503 08:12:12.242249 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 193 to 195, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 195 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:12.242185 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 193.14324, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.254173 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 193. Current term is 195. Ops: []
I20240503 08:12:12.254604 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 193.14324, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.254753 26567 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 195.14325 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14324
I20240503 08:12:12.262495 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 193. Current term is 196. Ops: []
I20240503 08:12:12.262820 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 193.14324, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.264820 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 195. Current term is 196. Ops: []
I20240503 08:12:12.265038 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.265591 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 193 LEADER]: Stepping down as leader of term 193
I20240503 08:12:12.265636 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 193 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.265710 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14324, Committed index: 14324, Last appended: 193.14324, Last appended by leader: 14324, Current term: 193, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.265822 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 193 FOLLOWER]: Advancing to term 194
I20240503 08:12:12.270562 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 194 FOLLOWER]: Advancing to term 195
I20240503 08:12:12.271224 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 195 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 193 index: 14324. Preceding OpId from leader: term: 195 index: 14325. (index mismatch)
I20240503 08:12:12.271497 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.277223 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.277282 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 196 FOLLOWER]: Advancing to term 197
I20240503 08:12:12.277846 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.277992 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 197 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.278177 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 197 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.278185 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 197 candidate_status { last_received { term: 193 index: 14324 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.278460 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 197 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.278553 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Leader election lost for term 197. Reason: could not achieve majority
I20240503 08:12:12.287154 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 195. Current term is 197. Ops: [195.14325-195.14325]
I20240503 08:12:12.287454 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 193.14324, Next index: 14325, Last known committed idx: 14324, Time since last communication: 0.000s
I20240503 08:12:12.294255 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 LEADER]: Stepping down as leader of term 195
I20240503 08:12:12.294325 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.294405 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14325, Committed index: 14325, Last appended: 195.14325, Last appended by leader: 14325, Current term: 195, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.294513 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 195 FOLLOWER]: Advancing to term 196
I20240503 08:12:12.303534 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 195 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:12.303624 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 195 FOLLOWER]: Advancing to term 196
I20240503 08:12:12.304252 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 196 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.304430 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 196 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.304690 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 196 candidate_status { last_received { term: 195 index: 14325 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.304673 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 196 candidate_status { last_received { term: 195 index: 14325 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.304827 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 196. Current term is 197.
I20240503 08:12:12.305011 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 196 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 196. Current term is 197.
I20240503 08:12:12.305053 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 196 election: Cancelling election due to peer responding with higher term
I20240503 08:12:12.305124 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 196 FOLLOWER]: Advancing to term 197
I20240503 08:12:12.305433 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 196 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 196.
I20240503 08:12:12.305802 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 197 FOLLOWER]: Leader election lost for term 196. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 196. Current term is 197.
I20240503 08:12:12.338490 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 197 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.338593 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 197 FOLLOWER]: Advancing to term 198
I20240503 08:12:12.339282 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 198 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.339448 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 198 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.339692 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 198 candidate_status { last_received { term: 195 index: 14325 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.339690 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 198 candidate_status { last_received { term: 195 index: 14325 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.339793 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 197 FOLLOWER]: Advancing to term 198
I20240503 08:12:12.339813 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 196 FOLLOWER]: Advancing to term 198
I20240503 08:12:12.340323 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 198 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 198.
I20240503 08:12:12.340376 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 198 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 198.
I20240503 08:12:12.340519 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 198 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:12.340618 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 198 FOLLOWER]: Leader election won for term 198
I20240503 08:12:12.340786 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 198 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.340878 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14325, Committed index: 14325, Last appended: 195.14325, Last appended by leader: 14325, Current term: 198, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.341439 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 195 to 198, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 198 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.371446 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 198 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 195 index: 14325. Preceding OpId from leader: term: 198 index: 14326. (index mismatch)
I20240503 08:12:12.371696 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14326, Last known committed idx: 14325, Time since last communication: 0.000s
I20240503 08:12:12.375775 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 198 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 193 index: 14324. Preceding OpId from leader: term: 198 index: 14326. (index mismatch)
I20240503 08:12:12.376027 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14326, Last known committed idx: 14324, Time since last communication: 0.000s
W20240503 08:12:12.393322 26539 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 196 current term: 198 is higher.
I20240503 08:12:12.407099 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 198 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:12.407204 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 198 FOLLOWER]: Advancing to term 199
I20240503 08:12:12.407992 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 199 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.408255 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 199 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.408332 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 199 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.408447 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 199 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.408641 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 199 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.408789 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 199 FOLLOWER]: Leader election lost for term 199. Reason: could not achieve majority
I20240503 08:12:12.411984 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 199 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 198. Current term is 199. Ops: []
I20240503 08:12:12.412204 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 198.14326, Next index: 14327, Last known committed idx: 14326, Time since last communication: 0.000s
I20240503 08:12:12.446382 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 199 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.446484 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 199 FOLLOWER]: Advancing to term 200
I20240503 08:12:12.447245 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.447499 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 200 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.447544 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 200 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.447618 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 200 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.448604 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 200 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.448604 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 198. Current term is 200. Ops: []
I20240503 08:12:12.448776 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Leader election lost for term 200. Reason: could not achieve majority
I20240503 08:12:12.448948 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 198.14326, Next index: 14327, Last known committed idx: 14326, Time since last communication: 0.000s
I20240503 08:12:12.459687 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 198 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:12.459770 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 198 FOLLOWER]: Advancing to term 199
I20240503 08:12:12.460386 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 199 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.460523 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 199 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.460731 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 199 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.460733 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 199 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.460837 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 199. Current term is 200.
I20240503 08:12:12.460992 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 199 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 199. Current term is 200.
I20240503 08:12:12.461066 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 199 election: Cancelling election due to peer responding with higher term
I20240503 08:12:12.461144 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 199 FOLLOWER]: Advancing to term 200
I20240503 08:12:12.461687 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 200 FOLLOWER]: Leader election lost for term 199. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 199. Current term is 200.
I20240503 08:12:12.465354 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 200 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 198. Current term is 200. Ops: []
I20240503 08:12:12.465569 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 198.14326, Next index: 14327, Last known committed idx: 14326, Time since last communication: 0.000s
I20240503 08:12:12.478497 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 198. Current term is 200. Ops: []
I20240503 08:12:12.478811 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 198.14326, Next index: 14327, Last known committed idx: 14326, Time since last communication: 0.000s
I20240503 08:12:12.482136 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 198 LEADER]: Stepping down as leader of term 198
I20240503 08:12:12.482192 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 198 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.482239 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14326, Committed index: 14326, Last appended: 198.14326, Last appended by leader: 14326, Current term: 198, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.482353 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 198 FOLLOWER]: Advancing to term 199
I20240503 08:12:12.490928 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.491003 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 200 FOLLOWER]: Advancing to term 201
I20240503 08:12:12.491583 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 201 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.491720 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 201 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.491910 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 201 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.491911 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 201 candidate_status { last_received { term: 198 index: 14326 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.492005 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 199 FOLLOWER]: Advancing to term 201
I20240503 08:12:12.492043 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 200 FOLLOWER]: Advancing to term 201
I20240503 08:12:12.492558 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 201 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 201.
I20240503 08:12:12.492575 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 201 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 201.
I20240503 08:12:12.492753 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 201 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:12.492870 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 201 FOLLOWER]: Leader election won for term 201
I20240503 08:12:12.492934 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 201 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:12.492990 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14326, Committed index: 14326, Last appended: 198.14326, Last appended by leader: 14326, Current term: 201, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.493414 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 198 to 201, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 201 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.523185 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 201 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 198 index: 14326. Preceding OpId from leader: term: 201 index: 14327. (index mismatch)
I20240503 08:12:12.523478 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14327, Last known committed idx: 14326, Time since last communication: 0.000s
I20240503 08:12:12.524931 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 201 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 198 index: 14326. Preceding OpId from leader: term: 201 index: 14327. (index mismatch)
I20240503 08:12:12.525118 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14327, Last known committed idx: 14326, Time since last communication: 0.000s
I20240503 08:12:12.554498 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 201 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:12.554600 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 201 FOLLOWER]: Advancing to term 202
I20240503 08:12:12.555357 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 202 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.555574 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 202 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.555655 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 202 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.555703 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 202 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.555889 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 202 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:12.555989 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 202 FOLLOWER]: Leader election lost for term 202. Reason: could not achieve majority
I20240503 08:12:12.559217 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 202 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 201. Current term is 202. Ops: []
I20240503 08:12:12.559460 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.585559 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 202 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 201. Current term is 202. Ops: []
I20240503 08:12:12.585829 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.587700 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 202 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.587754 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 202 FOLLOWER]: Advancing to term 203
I20240503 08:12:12.588384 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.588516 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 203 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.588701 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 203 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.588698 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 203 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.588946 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 203 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:12.589042 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Leader election lost for term 203. Reason: could not achieve majority
I20240503 08:12:12.609997 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 201. Current term is 203. Ops: []
I20240503 08:12:12.610283 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.612360 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 201 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:12.612421 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 201 FOLLOWER]: Advancing to term 202
I20240503 08:12:12.613087 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 202 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.613245 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 202 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.613425 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 202 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.613435 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 202 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.613502 26285 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 202. Current term is 203.
I20240503 08:12:12.613636 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 202 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 202. Current term is 203.
I20240503 08:12:12.613672 26173 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 202 election: Cancelling election due to peer responding with higher term
I20240503 08:12:12.613744 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 202 FOLLOWER]: Advancing to term 203
I20240503 08:12:12.614262 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 203 FOLLOWER]: Leader election lost for term 202. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 202. Current term is 203.
I20240503 08:12:12.614516 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 203 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 201. Current term is 203. Ops: []
I20240503 08:12:12.614739 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.636520 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 201. Current term is 203. Ops: []
I20240503 08:12:12.636783 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.647883 26516 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 201 LEADER]: Stepping down as leader of term 201
I20240503 08:12:12.647976 26516 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 201 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:12.648037 26516 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14327, Committed index: 14327, Last appended: 201.14327, Last appended by leader: 14327, Current term: 201, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.648167 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 201 FOLLOWER]: Advancing to term 202
I20240503 08:12:12.654039 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.654125 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 203 FOLLOWER]: Advancing to term 204
I20240503 08:12:12.654841 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 204 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.655124 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 204 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.655259 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 204 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.655330 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 202 FOLLOWER]: Advancing to term 204
I20240503 08:12:12.655421 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 204 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.655508 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 203 FOLLOWER]: Advancing to term 204
I20240503 08:12:12.656085 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 204 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 204.
I20240503 08:12:12.656164 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 204 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 204.
I20240503 08:12:12.656291 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 204 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:12.656397 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 204 FOLLOWER]: Leader election won for term 204
I20240503 08:12:12.656484 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 204 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.656555 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14327, Committed index: 14327, Last appended: 201.14327, Last appended by leader: 14327, Current term: 204, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.657171 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 201 to 204, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 204 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.684691 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 204 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 201 index: 14327. Preceding OpId from leader: term: 204 index: 14328. (index mismatch)
I20240503 08:12:12.685039 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.689496 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 204 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.689571 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 204 FOLLOWER]: Advancing to term 205
I20240503 08:12:12.690212 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.690336 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 204. Current term is 205. Ops: []
I20240503 08:12:12.690369 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 205 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.690553 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 205 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.690589 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 205 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.690737 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.690896 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 205 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.690990 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Leader election lost for term 205. Reason: could not achieve majority
I20240503 08:12:12.711300 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 204. Current term is 205. Ops: [204.14328-204.14328]
I20240503 08:12:12.711566 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.716380 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 204 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:12.716459 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 204 FOLLOWER]: Advancing to term 205
I20240503 08:12:12.717113 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 205 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.717291 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 205 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.717476 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 205 candidate_status { last_received { term: 204 index: 14328 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.717522 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 205 candidate_status { last_received { term: 204 index: 14328 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.717568 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 205: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:12.717731 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 205 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 204. Current term is 205. Ops: []
I20240503 08:12:12.717883 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 205 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.717970 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 205 FOLLOWER]: Leader election lost for term 205. Reason: could not achieve majority
I20240503 08:12:12.717958 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 204.14328, Next index: 14329, Last known committed idx: 14328, Time since last communication: 0.000s
I20240503 08:12:12.728634 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.728713 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 205 FOLLOWER]: Advancing to term 206
I20240503 08:12:12.729346 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 206 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.729492 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 206 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.729709 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 206 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.729709 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 206 candidate_status { last_received { term: 201 index: 14327 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.729822 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 205 FOLLOWER]: Advancing to term 206
I20240503 08:12:12.730474 26210 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 206 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 206 because replica has last-logged OpId of term: 204 index: 14328, which is greater than that of the candidate, which has last-logged OpId of term: 201 index: 14327.
I20240503 08:12:12.730633 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 206 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.730723 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 206 FOLLOWER]: Leader election lost for term 206. Reason: could not achieve majority
I20240503 08:12:12.733331 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 206 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 204. Current term is 206. Ops: [204.14328-204.14328]
I20240503 08:12:12.733552 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.749171 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 206 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.749293 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 206 FOLLOWER]: Advancing to term 207
I20240503 08:12:12.749927 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.750051 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 204. Current term is 207. Ops: []
I20240503 08:12:12.750092 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 207 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.750232 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 207 candidate_status { last_received { term: 204 index: 14328 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.750250 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 207 candidate_status { last_received { term: 204 index: 14328 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.750308 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 206 FOLLOWER]: Advancing to term 207
I20240503 08:12:12.750314 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 204.14328, Next index: 14329, Last known committed idx: 14328, Time since last communication: 0.000s
I20240503 08:12:12.751098 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 207 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 207.
I20240503 08:12:12.751334 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 207 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.751454 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 FOLLOWER]: Leader election won for term 207
I20240503 08:12:12.751540 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.751591 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14328, Committed index: 14328, Last appended: 204.14328, Last appended by leader: 14328, Current term: 207, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.752103 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 204 to 207, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 207 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:12.756915 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 207 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 204. Current term is 207. Ops: [204.14328-204.14328]
I20240503 08:12:12.757146 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 201.14327, Next index: 14328, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.769258 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 204 LEADER]: Stepping down as leader of term 204
I20240503 08:12:12.769348 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 204 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.769423 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14328, Committed index: 14328, Last appended: 204.14328, Last appended by leader: 14328, Current term: 204, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.769533 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 204 FOLLOWER]: Advancing to term 205
I20240503 08:12:12.781867 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 205 FOLLOWER]: Advancing to term 207
I20240503 08:12:12.782548 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 207 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 204 index: 14328. Preceding OpId from leader: term: 207 index: 14329. (index mismatch)
I20240503 08:12:12.782824 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14329, Last known committed idx: 14328, Time since last communication: 0.000s
I20240503 08:12:12.788659 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 207 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 201 index: 14327. Preceding OpId from leader: term: 207 index: 14329. (index mismatch)
I20240503 08:12:12.788915 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14329, Last known committed idx: 14327, Time since last communication: 0.000s
I20240503 08:12:12.819937 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 207 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:12.820040 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 207 FOLLOWER]: Advancing to term 208
I20240503 08:12:12.820731 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.820905 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 208 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.821069 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 208 candidate_status { last_received { term: 207 index: 14329 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.821131 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 208 candidate_status { last_received { term: 207 index: 14329 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.821305 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 208 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.821403 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Leader election lost for term 208. Reason: could not achieve majority
I20240503 08:12:12.822027 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 207. Current term is 208. Ops: []
I20240503 08:12:12.822223 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 207.14329, Next index: 14330, Last known committed idx: 14329, Time since last communication: 0.000s
I20240503 08:12:12.846508 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 207. Current term is 208. Ops: []
I20240503 08:12:12.846789 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 207.14329, Next index: 14330, Last known committed idx: 14329, Time since last communication: 0.000s
W20240503 08:12:12.856817 26349 lock_manager.cc:494] Txn 6 has not acquired the partition lock after 1000ms
I20240503 08:12:12.858014 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 207. Current term is 208. Ops: [207.14330-207.14330]
I20240503 08:12:12.858230 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 207.14329, Next index: 14330, Last known committed idx: 14329, Time since last communication: 0.000s
I20240503 08:12:12.863794 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.863864 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 208 FOLLOWER]: Advancing to term 209
I20240503 08:12:12.864696 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.864864 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 209 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.865002 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 209 candidate_status { last_received { term: 207 index: 14329 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.865002 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 209 candidate_status { last_received { term: 207 index: 14329 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.865242 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 209 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.865335 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Leader election lost for term 209. Reason: could not achieve majority
I20240503 08:12:12.889420 26567 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 207 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:12.889524 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 207 FOLLOWER]: Advancing to term 208
I20240503 08:12:12.890050 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 207. Current term is 209. Ops: [207.14330-207.14330]
I20240503 08:12:12.890295 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 207.14329, Next index: 14330, Last known committed idx: 14329, Time since last communication: 0.000s
I20240503 08:12:12.890493 26567 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 208 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.890677 26567 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 208 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.890794 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 208 candidate_status { last_received { term: 207 index: 14330 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.890812 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 208 candidate_status { last_received { term: 207 index: 14330 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.890861 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 208. Current term is 209.
I20240503 08:12:12.890957 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 208 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 207. Current term is 208. Ops: []
I20240503 08:12:12.891021 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 208 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 208. Current term is 209.
I20240503 08:12:12.891050 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 208 election: Cancelling election due to peer responding with higher term
I20240503 08:12:12.891110 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 208 FOLLOWER]: Advancing to term 209
I20240503 08:12:12.891278 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 207.14330, Next index: 14331, Last known committed idx: 14330, Time since last communication: 0.000s
I20240503 08:12:12.891726 26567 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 209 FOLLOWER]: Leader election lost for term 208. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 208. Current term is 209.
I20240503 08:12:12.892148 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 LEADER]: Stepping down as leader of term 207
I20240503 08:12:12.892186 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.892233 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14330, Committed index: 14330, Last appended: 207.14330, Last appended by leader: 14330, Current term: 207, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.892330 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 207 FOLLOWER]: Advancing to term 208
I20240503 08:12:12.919764 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 208 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.919884 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 208 FOLLOWER]: Advancing to term 209
I20240503 08:12:12.920568 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 209 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.920753 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 209 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:12.920958 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 209 candidate_status { last_received { term: 207 index: 14330 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.920962 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 209 candidate_status { last_received { term: 207 index: 14330 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.921080 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 209: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:12.921765 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 209 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 209.
I20240503 08:12:12.921939 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 209 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:12.922034 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 209 FOLLOWER]: Leader election won for term 209
I20240503 08:12:12.922107 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 209 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.922159 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14329, Committed index: 14329, Last appended: 207.14330, Last appended by leader: 14330, Current term: 209, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.922664 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 207 to 209. New cstate: current_term: 209 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:12.927278 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.927359 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 209 FOLLOWER]: Advancing to term 210
I20240503 08:12:12.928328 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 210 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.928485 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 210 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.928659 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 210 candidate_status { last_received { term: 207 index: 14329 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:12.928674 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 210 candidate_status { last_received { term: 207 index: 14329 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.928755 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 209 FOLLOWER]: Advancing to term 210
I20240503 08:12:12.929463 26285 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 210 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 210 because replica has last-logged OpId of term: 207 index: 14330, which is greater than that of the candidate, which has last-logged OpId of term: 207 index: 14329.
I20240503 08:12:12.929651 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 210 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:12.929739 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 210 FOLLOWER]: Leader election lost for term 210. Reason: could not achieve majority
I20240503 08:12:12.947095 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 210 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 209. Current term is 210. Ops: []
I20240503 08:12:12.947399 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14331, Last known committed idx: 14330, Time since last communication: 0.000s
I20240503 08:12:12.952940 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 210 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 209. Current term is 210. Ops: []
I20240503 08:12:12.953178 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14331, Last known committed idx: 14329, Time since last communication: 0.000s
W20240503 08:12:12.959213 26556 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 208 current term: 209 is higher.
I20240503 08:12:12.965238 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 210 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:12.965313 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 210 FOLLOWER]: Advancing to term 211
I20240503 08:12:12.965961 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.966125 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 211 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:12.966323 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 211 candidate_status { last_received { term: 207 index: 14330 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:12.966323 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 211 candidate_status { last_received { term: 207 index: 14330 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:12.966426 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 210 FOLLOWER]: Advancing to term 211
I20240503 08:12:12.967201 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 211 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 211.
I20240503 08:12:12.967386 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 211 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:12.967468 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 FOLLOWER]: Leader election won for term 211
I20240503 08:12:12.967520 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:12.967571 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14330, Committed index: 14330, Last appended: 207.14330, Last appended by leader: 14330, Current term: 211, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.968041 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 209 to 211, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 211 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:12.977077 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 209. Current term is 211. Ops: [209.14331-209.14331]
I20240503 08:12:12.977300 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 207.14330, Next index: 14331, Last known committed idx: 14330, Time since last communication: 0.000s
I20240503 08:12:12.977391 26378 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 211.14331 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14330
I20240503 08:12:12.981998 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 211 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 209. Current term is 211. Ops: [207.14330-209.14331]
I20240503 08:12:12.982239 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 207.14329, Next index: 14330, Last known committed idx: 14329, Time since last communication: 0.000s
W20240503 08:12:12.984386 26556 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 208 current term: 209 is higher.
I20240503 08:12:12.992030 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 211 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 209. Current term is 211. Ops: [207.14330-209.14331]
I20240503 08:12:12.992053 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 209. Current term is 211. Ops: [209.14331-209.14331]
I20240503 08:12:12.992305 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 207.14330, Next index: 14331, Last known committed idx: 14330, Time since last communication: 0.000s
I20240503 08:12:12.992468 26378 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 211.14331 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14330
I20240503 08:12:12.992666 26210 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 209 LEADER]: Stepping down as leader of term 209
I20240503 08:12:12.992703 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 207.14329, Next index: 14330, Last known committed idx: 14329, Time since last communication: 0.000s
I20240503 08:12:12.992736 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 211 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 207 index: 14329. Preceding OpId from leader: term: 211 index: 14332. (index mismatch)
I20240503 08:12:12.992722 26210 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 209 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:12.992913 26210 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14329, Committed index: 14329, Last appended: 209.14331, Last appended by leader: 14331, Current term: 209, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:12.993003 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 209 FOLLOWER]: Advancing to term 211
I20240503 08:12:12.993738 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 211 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 209 index: 14331. Preceding OpId from leader: term: 211 index: 14332. (index mismatch)
I20240503 08:12:12.993956 26567 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14331, Last known committed idx: 14329, Time since last communication: 0.000s
I20240503 08:12:12.994066 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14331, Last known committed idx: 14329, Time since last communication: 0.000s
I20240503 08:12:12.994139 26517 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 209.14331 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14329
I20240503 08:12:12.994366 26210 raft_consensus.cc:1203] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 211 FOLLOWER]: Deduplicated request from leader. Original: 207.14329->[207.14330-211.14332] Dedup: 207.14330->[211.14331-211.14332]
I20240503 08:12:12.994426 26210 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting all ops after (but not including) 14330
I20240503 08:12:12.994444 26210 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting uncommitted NO_OP operation due to leader change: 209.14331
I20240503 08:12:12.994463 26210 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 211 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:13.026286 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 211 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:13.026394 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 211 FOLLOWER]: Advancing to term 212
I20240503 08:12:13.027264 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 212 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.027397 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 212 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 211. Current term is 212. Ops: []
I20240503 08:12:13.027602 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 212 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.027684 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 212 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.027700 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 211.14332, Next index: 14333, Last known committed idx: 14332, Time since last communication: 0.000s
I20240503 08:12:13.027438 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 212 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.027869 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 212 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.027983 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 212 FOLLOWER]: Leader election lost for term 212. Reason: could not achieve majority
I20240503 08:12:13.062675 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 212 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 211. Current term is 212. Ops: []
I20240503 08:12:13.063005 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 211.14332, Next index: 14333, Last known committed idx: 14332, Time since last communication: 0.000s
I20240503 08:12:13.073200 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 212 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.073302 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 212 FOLLOWER]: Advancing to term 213
I20240503 08:12:13.073983 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.074162 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 213 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.074399 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 213 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.074399 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 213 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.074777 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 213 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.074887 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Leader election lost for term 213. Reason: could not achieve majority
I20240503 08:12:13.098592 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 211. Current term is 213. Ops: []
I20240503 08:12:13.098920 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 211.14332, Next index: 14333, Last known committed idx: 14332, Time since last communication: 0.000s
I20240503 08:12:13.101683 26567 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 LEADER]: Stepping down as leader of term 211
I20240503 08:12:13.101769 26567 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.101858 26567 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14332, Committed index: 14332, Last appended: 211.14332, Last appended by leader: 14332, Current term: 211, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.101962 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 211 FOLLOWER]: Advancing to term 212
I20240503 08:12:13.106143 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 211 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:13.106218 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 211 FOLLOWER]: Advancing to term 212
I20240503 08:12:13.106916 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 212 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.107079 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 212 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.107270 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 212 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.107273 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 212 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.107400 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 212. Current term is 213.
I20240503 08:12:13.107594 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 212 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 212. Current term is 213.
I20240503 08:12:13.107640 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 212 election: Cancelling election due to peer responding with higher term
I20240503 08:12:13.107695 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 212 FOLLOWER]: Advancing to term 213
I20240503 08:12:13.108043 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 212 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 212.
I20240503 08:12:13.108321 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 213 FOLLOWER]: Leader election lost for term 212. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 212. Current term is 213.
I20240503 08:12:13.131850 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.131968 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 213 FOLLOWER]: Advancing to term 214
I20240503 08:12:13.132683 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 214 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.132869 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 214 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.133090 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 214 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.133196 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 213 FOLLOWER]: Advancing to term 214
I20240503 08:12:13.133091 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 214 candidate_status { last_received { term: 211 index: 14332 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.133247 26285 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 212 FOLLOWER]: Advancing to term 214
I20240503 08:12:13.133816 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 214 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 214.
I20240503 08:12:13.133817 26285 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 214 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 214.
I20240503 08:12:13.134109 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 214 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:13.134227 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 214 FOLLOWER]: Leader election won for term 214
I20240503 08:12:13.134305 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 214 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.134382 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14332, Committed index: 14332, Last appended: 211.14332, Last appended by leader: 14332, Current term: 214, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.134828 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 211 to 214, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 214 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.163379 26285 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 214 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 211 index: 14332. Preceding OpId from leader: term: 214 index: 14333. (index mismatch)
I20240503 08:12:13.163671 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14333, Last known committed idx: 14332, Time since last communication: 0.000s
I20240503 08:12:13.165479 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 214 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 211 index: 14332. Preceding OpId from leader: term: 214 index: 14333. (index mismatch)
I20240503 08:12:13.165715 26516 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14333, Last known committed idx: 14332, Time since last communication: 0.000s
I20240503 08:12:13.195147 26567 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 214 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:13.195276 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 214 FOLLOWER]: Advancing to term 215
I20240503 08:12:13.195982 26567 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.196161 26567 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 215 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.196394 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 215 candidate_status { last_received { term: 214 index: 14333 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.196468 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 214 FOLLOWER]: Advancing to term 215
I20240503 08:12:13.196394 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 215 candidate_status { last_received { term: 214 index: 14333 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.197070 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 215 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 215.
I20240503 08:12:13.197247 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 215 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:13.197351 26567 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 FOLLOWER]: Leader election won for term 215
I20240503 08:12:13.197408 26567 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.197465 26567 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14333, Committed index: 14333, Last appended: 214.14333, Last appended by leader: 14333, Current term: 215, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.197921 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 214 to 215, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 215 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.200040 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 214. Current term is 215. Ops: []
I20240503 08:12:13.200246 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 214.14333, Next index: 14334, Last known committed idx: 14333, Time since last communication: 0.000s
I20240503 08:12:13.200326 26515 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 215.14334 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14333
I20240503 08:12:13.201112 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 215 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 214. Current term is 215. Ops: []
I20240503 08:12:13.201311 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 214.14333, Next index: 14334, Last known committed idx: 14333, Time since last communication: 0.000s
I20240503 08:12:13.227037 26285 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 214. Current term is 215. Ops: []
I20240503 08:12:13.227309 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 214.14333, Next index: 14334, Last known committed idx: 14333, Time since last communication: 0.000s
I20240503 08:12:13.227398 26516 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 215.14334 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14333
I20240503 08:12:13.228389 26134 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 214 LEADER]: Stepping down as leader of term 214
I20240503 08:12:13.228442 26134 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 214 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.228500 26134 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14333, Committed index: 14333, Last appended: 214.14333, Last appended by leader: 14333, Current term: 214, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.228623 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 214 FOLLOWER]: Advancing to term 215
I20240503 08:12:13.229279 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 215 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 214 index: 14333. Preceding OpId from leader: term: 215 index: 14334. (index mismatch)
I20240503 08:12:13.229609 26567 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14334, Last known committed idx: 14333, Time since last communication: 0.000s
I20240503 08:12:13.229645 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 215 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 214 index: 14333. Preceding OpId from leader: term: 215 index: 14334. (index mismatch)
I20240503 08:12:13.229900 26567 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14334, Last known committed idx: 14333, Time since last communication: 0.000s
I20240503 08:12:13.261395 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 215 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:13.261504 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 215 FOLLOWER]: Advancing to term 216
I20240503 08:12:13.261534 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 215 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:13.261613 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 215 FOLLOWER]: Advancing to term 216
I20240503 08:12:13.262202 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 216 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.262223 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 216 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.262373 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 216 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.262380 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 216 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.262538 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 216 candidate_status { last_received { term: 215 index: 14334 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.262576 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 216 candidate_status { last_received { term: 215 index: 14334 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.262588 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 216 candidate_status { last_received { term: 215 index: 14334 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.262727 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 216 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 216: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:13.262573 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 216 candidate_status { last_received { term: 215 index: 14334 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.262800 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 216 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 216: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:13.262892 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 216 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.262953 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 216 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.262997 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 216 FOLLOWER]: Leader election lost for term 216. Reason: could not achieve majority
I20240503 08:12:13.263017 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 216 FOLLOWER]: Leader election lost for term 216. Reason: could not achieve majority
I20240503 08:12:13.264168 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 216 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 215. Current term is 216. Ops: []
I20240503 08:12:13.264391 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 215.14334, Next index: 14335, Last known committed idx: 14334, Time since last communication: 0.000s
I20240503 08:12:13.266813 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 216 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 215. Current term is 216. Ops: []
I20240503 08:12:13.267019 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 215.14334, Next index: 14335, Last known committed idx: 14334, Time since last communication: 0.000s
I20240503 08:12:13.289546 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 216 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 215. Current term is 216. Ops: []
I20240503 08:12:13.289845 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 215.14334, Next index: 14335, Last known committed idx: 14334, Time since last communication: 0.000s
I20240503 08:12:13.295595 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 216 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.295702 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 216 FOLLOWER]: Advancing to term 217
I20240503 08:12:13.296406 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.296559 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 217 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.296713 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 217 candidate_status { last_received { term: 215 index: 14334 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.296716 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 217 candidate_status { last_received { term: 215 index: 14334 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.296782 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 216 FOLLOWER]: Advancing to term 217
I20240503 08:12:13.297334 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 217 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 217.
I20240503 08:12:13.297514 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 217 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.297614 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 FOLLOWER]: Leader election won for term 217
I20240503 08:12:13.297672 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:13.297724 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14334, Committed index: 14334, Last appended: 215.14334, Last appended by leader: 14334, Current term: 217, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.298213 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 215 to 217, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 217 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:13.302389 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 215. Current term is 217. Ops: []
I20240503 08:12:13.302604 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 215.14334, Next index: 14335, Last known committed idx: 14334, Time since last communication: 0.000s
I20240503 08:12:13.302714 26539 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 217.14335 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14334
I20240503 08:12:13.307159 26567 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 LEADER]: Stepping down as leader of term 215
I20240503 08:12:13.307222 26567 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.307286 26567 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14334, Committed index: 14334, Last appended: 215.14334, Last appended by leader: 14334, Current term: 215, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.307404 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 215 FOLLOWER]: Advancing to term 216
I20240503 08:12:13.320555 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 216 FOLLOWER]: Advancing to term 217
I20240503 08:12:13.321549 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 217 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 215 index: 14334. Preceding OpId from leader: term: 217 index: 14335. (index mismatch)
I20240503 08:12:13.321813 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14335, Last known committed idx: 14334, Time since last communication: 0.000s
I20240503 08:12:13.323065 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 217 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 215 index: 14334. Preceding OpId from leader: term: 217 index: 14335. (index mismatch)
I20240503 08:12:13.323239 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14335, Last known committed idx: 14334, Time since last communication: 0.000s
I20240503 08:12:13.380935 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 217 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:13.381042 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 217 FOLLOWER]: Advancing to term 218
I20240503 08:12:13.381714 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 218 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.381887 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 218 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.382103 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 218 candidate_status { last_received { term: 217 index: 14335 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.382107 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 218 candidate_status { last_received { term: 217 index: 14335 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.382347 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 218 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.382462 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 218 FOLLOWER]: Leader election lost for term 218. Reason: could not achieve majority
I20240503 08:12:13.383405 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 218 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 217. Current term is 218. Ops: []
I20240503 08:12:13.383610 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 217.14335, Next index: 14336, Last known committed idx: 14335, Time since last communication: 0.000s
I20240503 08:12:13.397039 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 LEADER]: Stepping down as leader of term 217
I20240503 08:12:13.397140 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:13.397233 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14335, Committed index: 14335, Last appended: 217.14335, Last appended by leader: 14335, Current term: 217, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.397344 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 217 FOLLOWER]: Advancing to term 218
I20240503 08:12:13.406188 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 217 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:13.406276 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 217 FOLLOWER]: Advancing to term 218
I20240503 08:12:13.407038 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 218 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.407223 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 218 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.407411 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 218 candidate_status { last_received { term: 217 index: 14335 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.407411 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 218 candidate_status { last_received { term: 217 index: 14335 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.407532 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 218 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 218: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:13.408063 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 218 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 218.
I20240503 08:12:13.408218 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 218 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:13.408339 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 218 FOLLOWER]: Leader election won for term 218
I20240503 08:12:13.408392 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 218 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.408443 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14335, Committed index: 14335, Last appended: 217.14335, Last appended by leader: 14335, Current term: 218, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.408854 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 217 to 218, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 218 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.415149 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 218 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.415218 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 218 FOLLOWER]: Advancing to term 219
I20240503 08:12:13.415794 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.415942 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 219 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.416105 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 219 candidate_status { last_received { term: 217 index: 14335 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.416127 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 219 candidate_status { last_received { term: 217 index: 14335 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.416191 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 218 FOLLOWER]: Advancing to term 219
I20240503 08:12:13.416723 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 219 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 219.
I20240503 08:12:13.416903 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 219 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.416991 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 FOLLOWER]: Leader election won for term 219
I20240503 08:12:13.417049 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.417104 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14335, Committed index: 14335, Last appended: 217.14335, Last appended by leader: 14335, Current term: 219, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.425685 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 218 to 219, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 219 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:13.432529 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 219 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 218. Current term is 219. Ops: []
I20240503 08:12:13.432818 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14336, Last known committed idx: 14335, Time since last communication: 0.000s
I20240503 08:12:13.436504 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 218. Current term is 219. Ops: []
I20240503 08:12:13.436776 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14336, Last known committed idx: 14335, Time since last communication: 0.000s
I20240503 08:12:13.436890 26567 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 219.14336 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14335
I20240503 08:12:13.440173 26284 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 218 LEADER]: Stepping down as leader of term 218
I20240503 08:12:13.440232 26284 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 218 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.440294 26284 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14335, Committed index: 14335, Last appended: 218.14336, Last appended by leader: 14336, Current term: 218, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.440408 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 218 FOLLOWER]: Advancing to term 219
I20240503 08:12:13.441037 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 219 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 218 index: 14336. Preceding OpId from leader: term: 219 index: 14336. (term mismatch)
I20240503 08:12:13.441074 26284 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14335
I20240503 08:12:13.441087 26284 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted NO_OP operation due to leader change: 218.14336
I20240503 08:12:13.441102 26284 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 219 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:13.441402 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14336, Last known committed idx: 14335, Time since last communication: 0.000s
I20240503 08:12:13.446066 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 219 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 217 index: 14335. Preceding OpId from leader: term: 219 index: 14336. (index mismatch)
I20240503 08:12:13.446470 26502 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14336, Last known committed idx: 14335, Time since last communication: 0.000s
I20240503 08:12:13.477686 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 219 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:13.477782 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 219 FOLLOWER]: Advancing to term 220
I20240503 08:12:13.478461 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.478664 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 220 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.478828 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 220 candidate_status { last_received { term: 219 index: 14336 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.478833 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 220 candidate_status { last_received { term: 219 index: 14336 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.479112 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 220 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.479243 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Leader election lost for term 220. Reason: could not achieve majority
I20240503 08:12:13.484019 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 219. Current term is 220. Ops: []
I20240503 08:12:13.484272 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 219.14336, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.500619 26567 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 219 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:13.500731 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 219 FOLLOWER]: Advancing to term 220
I20240503 08:12:13.501462 26567 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 220 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.501706 26567 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 220 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.501773 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 220 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 219. Current term is 220. Ops: []
I20240503 08:12:13.501763 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 220 candidate_status { last_received { term: 219 index: 14336 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.501763 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 220 candidate_status { last_received { term: 219 index: 14336 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.502002 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 220: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:13.502009 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 219.14336, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.502166 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 220 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:13.502262 26567 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 220 FOLLOWER]: Leader election lost for term 220. Reason: could not achieve majority
I20240503 08:12:13.511086 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 219. Current term is 220. Ops: []
I20240503 08:12:13.511377 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 219.14336, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.519466 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.519564 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 220 FOLLOWER]: Advancing to term 221
I20240503 08:12:13.520229 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.520385 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 221 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.520584 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 221 candidate_status { last_received { term: 219 index: 14336 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.520591 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 221 candidate_status { last_received { term: 219 index: 14336 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.520674 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 220 FOLLOWER]: Advancing to term 221
I20240503 08:12:13.521250 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 221 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 221.
I20240503 08:12:13.521442 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 221 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:13.521535 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 FOLLOWER]: Leader election won for term 221
I20240503 08:12:13.521602 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:13.521658 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14336, Committed index: 14336, Last appended: 219.14336, Last appended by leader: 14336, Current term: 221, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.522085 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 219 to 221, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 221 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:13.534135 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 221 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 219. Current term is 221. Ops: []
I20240503 08:12:13.534396 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 219.14336, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.542953 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 219. Current term is 221. Ops: []
I20240503 08:12:13.543228 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 219.14336, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.543352 26516 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 221.14337 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14336
I20240503 08:12:13.547528 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 221 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 219 index: 14336. Preceding OpId from leader: term: 221 index: 14337. (index mismatch)
I20240503 08:12:13.547755 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.556751 26134 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 LEADER]: Stepping down as leader of term 219
I20240503 08:12:13.556829 26134 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.556902 26134 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14336, Committed index: 14336, Last appended: 219.14336, Last appended by leader: 14336, Current term: 219, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.557022 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 219 FOLLOWER]: Advancing to term 221
I20240503 08:12:13.557703 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 221 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 219 index: 14336. Preceding OpId from leader: term: 221 index: 14337. (index mismatch)
I20240503 08:12:13.557960 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14337, Last known committed idx: 14336, Time since last communication: 0.000s
I20240503 08:12:13.588776 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 221 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:13.588881 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 221 FOLLOWER]: Advancing to term 222
I20240503 08:12:13.589587 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 222 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.589756 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 222 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.589977 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 222 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.589987 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 222 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.590278 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 222 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.590392 26516 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 222 FOLLOWER]: Leader election lost for term 222. Reason: could not achieve majority
I20240503 08:12:13.593811 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 222 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 221. Current term is 222. Ops: []
I20240503 08:12:13.594091 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 221.14337, Next index: 14338, Last known committed idx: 14337, Time since last communication: 0.000s
I20240503 08:12:13.601826 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 221 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:13.601920 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 221 FOLLOWER]: Advancing to term 222
I20240503 08:12:13.602679 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 222 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.602824 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 222 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 221. Current term is 222. Ops: []
I20240503 08:12:13.602880 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 222 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.603032 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 221.14337, Next index: 14338, Last known committed idx: 14337, Time since last communication: 0.000s
I20240503 08:12:13.603041 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 222 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.603049 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 222 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.603168 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 222 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 222: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:13.603302 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 222 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:13.603397 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 222 FOLLOWER]: Leader election lost for term 222. Reason: could not achieve majority
I20240503 08:12:13.614826 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 LEADER]: Stepping down as leader of term 221
I20240503 08:12:13.614925 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:13.614992 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14337, Committed index: 14337, Last appended: 221.14337, Last appended by leader: 14337, Current term: 221, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.615120 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 221 FOLLOWER]: Advancing to term 222
I20240503 08:12:13.626266 26516 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 222 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.626382 26516 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 222 FOLLOWER]: Advancing to term 223
I20240503 08:12:13.627328 26516 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 223 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.627526 26516 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 223 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.627763 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 223 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.627763 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 223 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.627871 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 222 FOLLOWER]: Advancing to term 223
I20240503 08:12:13.627874 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 222 FOLLOWER]: Advancing to term 223
I20240503 08:12:13.628592 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 223 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 223.
I20240503 08:12:13.628607 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 223 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 223.
I20240503 08:12:13.629004 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 223 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:13.629118 26516 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 223 FOLLOWER]: Leader election won for term 223
I20240503 08:12:13.629164 26516 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 223 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.629222 26516 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14337, Committed index: 14337, Last appended: 221.14337, Last appended by leader: 14337, Current term: 223, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.629709 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 221 to 223, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 223 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:13.643107 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 220 current term: 223 is higher.
I20240503 08:12:13.662338 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 223 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.662431 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 223 FOLLOWER]: Advancing to term 224
I20240503 08:12:13.663220 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.663345 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 223. Current term is 224. Ops: []
I20240503 08:12:13.663388 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 224 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.663594 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 224 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.663602 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14338, Last known committed idx: 14337, Time since last communication: 0.000s
I20240503 08:12:13.663676 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 223 FOLLOWER]: Advancing to term 224
I20240503 08:12:13.663627 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 224 candidate_status { last_received { term: 221 index: 14337 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.664448 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 224 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 224.
I20240503 08:12:13.664753 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 224 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:13.664858 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 FOLLOWER]: Leader election won for term 224
I20240503 08:12:13.664906 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.664968 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14337, Committed index: 14337, Last appended: 221.14337, Last appended by leader: 14337, Current term: 224, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.666146 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 224 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 223. Current term is 224. Ops: []
I20240503 08:12:13.666461 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14338, Last known committed idx: 14337, Time since last communication: 0.000s
I20240503 08:12:13.670795 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 223 to 224, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 224 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:13.689522 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 221 current term: 223 is higher.
I20240503 08:12:13.691478 26134 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 223 LEADER]: Stepping down as leader of term 223
I20240503 08:12:13.691550 26134 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 223 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.691619 26134 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14337, Committed index: 14337, Last appended: 223.14338, Last appended by leader: 14338, Current term: 223, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.691736 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 223 FOLLOWER]: Advancing to term 224
I20240503 08:12:13.692531 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 224 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 223 index: 14338. Preceding OpId from leader: term: 224 index: 14338. (term mismatch)
I20240503 08:12:13.692577 26134 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14337
I20240503 08:12:13.692600 26134 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 223.14338
I20240503 08:12:13.692621 26134 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 224 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:13.692929 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14338, Last known committed idx: 14337, Time since last communication: 0.000s
I20240503 08:12:13.696996 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 224 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 221 index: 14337. Preceding OpId from leader: term: 224 index: 14338. (index mismatch)
I20240503 08:12:13.697274 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14338, Last known committed idx: 14337, Time since last communication: 0.000s
I20240503 08:12:13.724680 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 224 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:13.724809 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 224 FOLLOWER]: Advancing to term 225
I20240503 08:12:13.725673 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.725840 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 225 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.725965 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 225 candidate_status { last_received { term: 224 index: 14338 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.726016 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 225 candidate_status { last_received { term: 224 index: 14338 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.726179 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 225 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.726274 26502 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Leader election lost for term 225. Reason: could not achieve majority
I20240503 08:12:13.726368 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 224. Current term is 225. Ops: []
I20240503 08:12:13.726568 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 224.14338, Next index: 14339, Last known committed idx: 14338, Time since last communication: 0.000s
I20240503 08:12:13.727679 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 224 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:13.727723 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 224 FOLLOWER]: Advancing to term 225
I20240503 08:12:13.728305 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 225 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.728442 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 225 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.728595 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 225 candidate_status { last_received { term: 224 index: 14338 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.728628 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 225 candidate_status { last_received { term: 224 index: 14338 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.728689 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 225: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:13.728844 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 225 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.728930 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 225 FOLLOWER]: Leader election lost for term 225. Reason: could not achieve majority
I20240503 08:12:13.729604 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 225 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 224. Current term is 225. Ops: []
I20240503 08:12:13.729842 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 224.14338, Next index: 14339, Last known committed idx: 14338, Time since last communication: 0.000s
I20240503 08:12:13.748710 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 224. Current term is 225. Ops: []
I20240503 08:12:13.749163 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 224.14338, Next index: 14339, Last known committed idx: 14338, Time since last communication: 0.000s
I20240503 08:12:13.757427 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 LEADER]: Stepping down as leader of term 224
I20240503 08:12:13.757521 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:13.757597 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14338, Committed index: 14338, Last appended: 224.14338, Last appended by leader: 14338, Current term: 224, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.757720 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 224 FOLLOWER]: Advancing to term 225
I20240503 08:12:13.764912 26502 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.765012 26502 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 225 FOLLOWER]: Advancing to term 226
I20240503 08:12:13.765697 26502 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 226 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.765877 26502 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 226 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.766116 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 226 candidate_status { last_received { term: 224 index: 14338 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.766139 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 226 candidate_status { last_received { term: 224 index: 14338 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.766216 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 225 FOLLOWER]: Advancing to term 226
I20240503 08:12:13.766220 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 225 FOLLOWER]: Advancing to term 226
I20240503 08:12:13.766909 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 226 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 226.
I20240503 08:12:13.766916 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 226 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 226.
I20240503 08:12:13.767120 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 226 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:13.767236 26502 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 226 FOLLOWER]: Leader election won for term 226
I20240503 08:12:13.767285 26502 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 226 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.767376 26502 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14338, Committed index: 14338, Last appended: 224.14338, Last appended by leader: 14338, Current term: 226, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.767871 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 224 to 226, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 226 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:13.768764 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 224 current term: 226 is higher.
W20240503 08:12:13.790948 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 224 current term: 226 is higher.
I20240503 08:12:13.795374 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 226 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 224 index: 14338. Preceding OpId from leader: term: 226 index: 14339. (index mismatch)
I20240503 08:12:13.795648 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14339, Last known committed idx: 14338, Time since last communication: 0.000s
I20240503 08:12:13.797192 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 226 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 224 index: 14338. Preceding OpId from leader: term: 226 index: 14339. (index mismatch)
I20240503 08:12:13.797398 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14339, Last known committed idx: 14338, Time since last communication: 0.000s
I20240503 08:12:13.828311 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 226 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:13.828408 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 226 FOLLOWER]: Advancing to term 227
I20240503 08:12:13.829105 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 227 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.829257 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 227 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.829479 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 227 candidate_status { last_received { term: 226 index: 14339 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.829486 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 227 candidate_status { last_received { term: 226 index: 14339 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.829720 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 227 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.829838 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 227 FOLLOWER]: Leader election lost for term 227. Reason: could not achieve majority
I20240503 08:12:13.830857 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 227 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 226. Current term is 227. Ops: []
I20240503 08:12:13.831082 26516 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 226.14339, Next index: 14340, Last known committed idx: 14339, Time since last communication: 0.000s
I20240503 08:12:13.855319 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 227 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 226. Current term is 227. Ops: []
I20240503 08:12:13.855583 26502 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 226.14339, Next index: 14340, Last known committed idx: 14339, Time since last communication: 0.000s
I20240503 08:12:13.857087 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 226 LEADER]: Stepping down as leader of term 226
I20240503 08:12:13.857128 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 226 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:13.857184 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14339, Committed index: 14339, Last appended: 226.14339, Last appended by leader: 14339, Current term: 226, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.857275 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 226 FOLLOWER]: Advancing to term 227
I20240503 08:12:13.867931 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 227 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:13.868024 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 227 FOLLOWER]: Advancing to term 228
I20240503 08:12:13.868706 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 228 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.868878 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 228 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:13.869051 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 228 candidate_status { last_received { term: 226 index: 14339 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.869055 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 228 candidate_status { last_received { term: 226 index: 14339 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:13.869175 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 227 FOLLOWER]: Advancing to term 228
I20240503 08:12:13.869763 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 228.
I20240503 08:12:13.869943 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 228 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:13.870046 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 228 FOLLOWER]: Leader election won for term 228
I20240503 08:12:13.870105 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 228 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:13.870162 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14339, Committed index: 14339, Last appended: 226.14339, Last appended by leader: 14339, Current term: 228, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.870638 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 226 to 228, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 228 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:13.883183 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 226 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:13.883275 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 226 FOLLOWER]: Advancing to term 227
I20240503 08:12:13.884002 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 227 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:13.884156 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 227 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:13.884367 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 227 candidate_status { last_received { term: 226 index: 14339 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:13.884367 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 227 candidate_status { last_received { term: 226 index: 14339 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:13.884517 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 227. Current term is 228.
I20240503 08:12:13.884686 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 227 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 227. Current term is 228.
I20240503 08:12:13.884725 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 227 election: Cancelling election due to peer responding with higher term
I20240503 08:12:13.884799 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 227 FOLLOWER]: Advancing to term 228
I20240503 08:12:13.885407 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 228 FOLLOWER]: Leader election lost for term 227. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 227. Current term is 228.
I20240503 08:12:13.901481 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 226 index: 14339. Preceding OpId from leader: term: 228 index: 14340. (index mismatch)
I20240503 08:12:13.901739 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14340, Last known committed idx: 14339, Time since last communication: 0.000s
I20240503 08:12:13.907243 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 228 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 226 index: 14339. Preceding OpId from leader: term: 228 index: 14340. (index mismatch)
I20240503 08:12:13.907521 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14340, Last known committed idx: 14339, Time since last communication: 0.000s
W20240503 08:12:13.992337 26348 lock_manager.cc:494] Txn 7 has not acquired the partition lock after 1000ms
I20240503 08:12:14.068876 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 228 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.068995 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 228 FOLLOWER]: Advancing to term 229
I20240503 08:12:14.069833 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 229 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.069967 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 229 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 228. Current term is 229. Ops: []
I20240503 08:12:14.070010 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 229 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.070150 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 229 candidate_status { last_received { term: 228 index: 14342 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.070214 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 228.14342, Next index: 14343, Last known committed idx: 14342, Time since last communication: 0.000s
I20240503 08:12:14.070180 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 229 candidate_status { last_received { term: 228 index: 14342 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.070434 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 229 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.070531 26539 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 229 FOLLOWER]: Leader election lost for term 229. Reason: could not achieve majority
I20240503 08:12:14.103143 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 229 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 228. Current term is 229. Ops: []
I20240503 08:12:14.103540 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 228.14342, Next index: 14343, Last known committed idx: 14342, Time since last communication: 0.000s
I20240503 08:12:14.112439 26556 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 228 LEADER]: Stepping down as leader of term 228
I20240503 08:12:14.112524 26556 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 228 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:14.112583 26556 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14342, Committed index: 14342, Last appended: 228.14342, Last appended by leader: 14342, Current term: 228, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.112709 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 228 FOLLOWER]: Advancing to term 229
I20240503 08:12:14.115272 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 229 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.115340 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 229 FOLLOWER]: Advancing to term 230
I20240503 08:12:14.115880 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 230 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.116024 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 230 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.116225 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 230 candidate_status { last_received { term: 228 index: 14342 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.116214 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 230 candidate_status { last_received { term: 228 index: 14342 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.116359 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 229 FOLLOWER]: Advancing to term 230
I20240503 08:12:14.116863 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 230.
I20240503 08:12:14.117039 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 230 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.117141 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 230 FOLLOWER]: Leader election won for term 230
I20240503 08:12:14.117198 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 230 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:14.117254 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14342, Committed index: 14342, Last appended: 228.14342, Last appended by leader: 14342, Current term: 230, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.117698 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 228 to 230, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 230 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:14.143182 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.143291 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 228 FOLLOWER]: Advancing to term 229
I20240503 08:12:14.144028 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 229 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.144194 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 229 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.144418 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 229 candidate_status { last_received { term: 228 index: 14342 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.144428 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 229 candidate_status { last_received { term: 228 index: 14342 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.144528 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 229. Current term is 230.
I20240503 08:12:14.144691 26098 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 229 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 229. Current term is 230.
I20240503 08:12:14.144731 26098 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 229 election: Cancelling election due to peer responding with higher term
I20240503 08:12:14.144793 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 229 FOLLOWER]: Advancing to term 230
I20240503 08:12:14.145426 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 230 FOLLOWER]: Leader election lost for term 229. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 229. Current term is 230.
I20240503 08:12:14.151173 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 228 index: 14342. Preceding OpId from leader: term: 230 index: 14343. (index mismatch)
I20240503 08:12:14.151477 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14343, Last known committed idx: 14342, Time since last communication: 0.000s
I20240503 08:12:14.154125 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 230 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 228 index: 14342. Preceding OpId from leader: term: 230 index: 14343. (index mismatch)
I20240503 08:12:14.154407 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14343, Last known committed idx: 14342, Time since last communication: 0.000s
I20240503 08:12:14.182504 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:14.182646 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 230 FOLLOWER]: Advancing to term 231
I20240503 08:12:14.183377 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 231 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.183554 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 231 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 230. Current term is 231. Ops: []
I20240503 08:12:14.183557 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 231 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.183770 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 231 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.183822 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 231 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.183888 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 230.14343, Next index: 14344, Last known committed idx: 14343, Time since last communication: 0.000s
I20240503 08:12:14.183984 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 231 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.184069 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 231 FOLLOWER]: Leader election lost for term 231. Reason: could not achieve majority
I20240503 08:12:14.214179 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 231 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 230. Current term is 231. Ops: []
I20240503 08:12:14.214486 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 230.14343, Next index: 14344, Last known committed idx: 14343, Time since last communication: 0.000s
I20240503 08:12:14.218596 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 231 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.218689 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 231 FOLLOWER]: Advancing to term 232
I20240503 08:12:14.219355 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 232 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.219514 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 232 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.219667 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 232 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.219717 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 232 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.219928 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 232 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.220012 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 232 FOLLOWER]: Leader election lost for term 232. Reason: could not achieve majority
I20240503 08:12:14.245359 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 232 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 230. Current term is 232. Ops: []
I20240503 08:12:14.245784 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 230.14343, Next index: 14344, Last known committed idx: 14343, Time since last communication: 0.000s
I20240503 08:12:14.266912 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 230 LEADER]: Stepping down as leader of term 230
I20240503 08:12:14.267000 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 230 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:14.267067 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14343, Committed index: 14343, Last appended: 230.14343, Last appended by leader: 14343, Current term: 230, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.267181 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 230 FOLLOWER]: Advancing to term 231
I20240503 08:12:14.279091 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 232 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.279177 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 232 FOLLOWER]: Advancing to term 233
I20240503 08:12:14.279857 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 233 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.280023 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 233 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.280225 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 233 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.280288 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 231 FOLLOWER]: Advancing to term 233
I20240503 08:12:14.280246 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 233 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.280845 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 233.
I20240503 08:12:14.281049 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 233 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.281148 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 233 FOLLOWER]: Leader election won for term 233
I20240503 08:12:14.281198 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 233 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:14.281340 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14343, Committed index: 14343, Last appended: 230.14343, Last appended by leader: 14343, Current term: 233, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.281786 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 230 to 233, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 233 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:14.294835 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 230 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:14.294932 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 230 FOLLOWER]: Advancing to term 231
I20240503 08:12:14.295565 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 231 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.295711 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 231 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.295933 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 231 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.295910 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 231 candidate_status { last_received { term: 230 index: 14343 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.296025 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 231. Current term is 233.
I20240503 08:12:14.296192 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 231 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 231. Current term is 233.
I20240503 08:12:14.296236 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 231 election: Cancelling election due to peer responding with higher term
I20240503 08:12:14.296308 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 231 FOLLOWER]: Advancing to term 233
I20240503 08:12:14.296833 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 233 FOLLOWER]: Leader election lost for term 231. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 231. Current term is 233.
I20240503 08:12:14.307801 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 233 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 230 index: 14343. Preceding OpId from leader: term: 233 index: 14344. (index mismatch)
I20240503 08:12:14.308094 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14344, Last known committed idx: 14343, Time since last communication: 0.000s
I20240503 08:12:14.318202 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 230 index: 14343. Preceding OpId from leader: term: 233 index: 14344. (index mismatch)
I20240503 08:12:14.318495 26556 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14344, Last known committed idx: 14343, Time since last communication: 0.000s
I20240503 08:12:14.364014 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 233 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.364120 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 233 FOLLOWER]: Advancing to term 234
I20240503 08:12:14.364940 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.365062 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 233. Current term is 234. Ops: []
I20240503 08:12:14.365198 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 234 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.365267 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 234 candidate_status { last_received { term: 233 index: 14344 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.365281 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 234 candidate_status { last_received { term: 233 index: 14344 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.365281 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 233.14344, Next index: 14345, Last known committed idx: 14344, Time since last communication: 0.000s
I20240503 08:12:14.365578 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 234 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.365700 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Leader election lost for term 234. Reason: could not achieve majority
I20240503 08:12:14.389246 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 233. Current term is 234. Ops: []
I20240503 08:12:14.389539 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 233.14344, Next index: 14345, Last known committed idx: 14344, Time since last communication: 0.000s
I20240503 08:12:14.397996 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.398082 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 233 FOLLOWER]: Advancing to term 234
I20240503 08:12:14.398761 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 234 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.398905 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 234 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.399106 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 234 candidate_status { last_received { term: 233 index: 14344 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.399191 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 234: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:14.399111 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 234 candidate_status { last_received { term: 233 index: 14344 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.399350 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 234 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.399453 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 234 FOLLOWER]: Leader election lost for term 234. Reason: could not achieve majority
I20240503 08:12:14.402489 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 234 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 233. Current term is 234. Ops: []
I20240503 08:12:14.402727 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 233.14344, Next index: 14345, Last known committed idx: 14344, Time since last communication: 0.000s
I20240503 08:12:14.405578 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.405638 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 234 FOLLOWER]: Advancing to term 235
I20240503 08:12:14.406248 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 235 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.406598 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 235 candidate_status { last_received { term: 233 index: 14344 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.406563 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 235 candidate_status { last_received { term: 233 index: 14344 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.406725 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 234 FOLLOWER]: Advancing to term 235
I20240503 08:12:14.407296 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 235 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 235.
I20240503 08:12:14.406392 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 235 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.407466 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 235 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:14.407549 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 235 FOLLOWER]: Leader election won for term 235
I20240503 08:12:14.407656 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 235 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:14.407701 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14344, Committed index: 14344, Last appended: 233.14344, Last appended by leader: 14344, Current term: 235, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.408228 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 233 to 235, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 235 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:14.416556 26556 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 233 LEADER]: Stepping down as leader of term 233
I20240503 08:12:14.416638 26556 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 233 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:14.416702 26556 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14344, Committed index: 14344, Last appended: 233.14344, Last appended by leader: 14344, Current term: 233, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.416839 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 233 FOLLOWER]: Advancing to term 234
I20240503 08:12:14.430819 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 234 FOLLOWER]: Advancing to term 235
I20240503 08:12:14.431571 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 235 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 233 index: 14344. Preceding OpId from leader: term: 235 index: 14345. (index mismatch)
I20240503 08:12:14.431830 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14345, Last known committed idx: 14344, Time since last communication: 0.000s
I20240503 08:12:14.439657 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 235 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 233 index: 14344. Preceding OpId from leader: term: 235 index: 14345. (index mismatch)
I20240503 08:12:14.439946 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14345, Last known committed idx: 14344, Time since last communication: 0.000s
I20240503 08:12:14.463524 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 235 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:14.463636 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 235 FOLLOWER]: Advancing to term 236
I20240503 08:12:14.464323 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 236 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.464511 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 236 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.464735 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 236 candidate_status { last_received { term: 235 index: 14345 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.464740 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 236 candidate_status { last_received { term: 235 index: 14345 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.465077 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 236 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.465188 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 236 FOLLOWER]: Leader election lost for term 236. Reason: could not achieve majority
I20240503 08:12:14.468822 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 236 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 235. Current term is 236. Ops: []
I20240503 08:12:14.469029 26604 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 235.14345, Next index: 14346, Last known committed idx: 14345, Time since last communication: 0.000s
I20240503 08:12:14.471150 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 235 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:14.471201 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 235 FOLLOWER]: Advancing to term 236
I20240503 08:12:14.471866 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 236 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.471980 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 236 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 235. Current term is 236. Ops: []
I20240503 08:12:14.472021 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 236 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.472169 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 236 candidate_status { last_received { term: 235 index: 14345 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.472231 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 236 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 236: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:14.472218 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 236 candidate_status { last_received { term: 235 index: 14345 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.472255 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 235.14345, Next index: 14346, Last known committed idx: 14345, Time since last communication: 0.000s
I20240503 08:12:14.472519 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 236 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.472587 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 236 FOLLOWER]: Leader election lost for term 236. Reason: could not achieve majority
I20240503 08:12:14.477214 26604 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 235 LEADER]: Stepping down as leader of term 235
I20240503 08:12:14.477280 26604 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 235 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:14.477361 26604 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14345, Committed index: 14345, Last appended: 235.14345, Last appended by leader: 14345, Current term: 235, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.477465 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 235 FOLLOWER]: Advancing to term 236
I20240503 08:12:14.495730 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 236 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.495831 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 236 FOLLOWER]: Advancing to term 237
I20240503 08:12:14.496564 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 237 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.496737 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 237 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.496975 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 237 candidate_status { last_received { term: 235 index: 14345 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.496975 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 237 candidate_status { last_received { term: 235 index: 14345 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.497097 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 236 FOLLOWER]: Advancing to term 237
I20240503 08:12:14.497102 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 236 FOLLOWER]: Advancing to term 237
I20240503 08:12:14.497802 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 237 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 237.
I20240503 08:12:14.497815 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 237 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 237.
I20240503 08:12:14.498020 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 237 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:14.498113 26556 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 237 FOLLOWER]: Leader election won for term 237
I20240503 08:12:14.498175 26556 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 237 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:14.498217 26556 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14345, Committed index: 14345, Last appended: 235.14345, Last appended by leader: 14345, Current term: 237, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.498770 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 235 to 237, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 237 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:14.530222 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 237 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 235 index: 14345. Preceding OpId from leader: term: 237 index: 14346. (index mismatch)
I20240503 08:12:14.530557 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14346, Last known committed idx: 14345, Time since last communication: 0.000s
I20240503 08:12:14.530865 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 237 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 235 index: 14345. Preceding OpId from leader: term: 237 index: 14346. (index mismatch)
I20240503 08:12:14.531044 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14346, Last known committed idx: 14345, Time since last communication: 0.000s
I20240503 08:12:14.561712 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 237 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.561823 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 237 FOLLOWER]: Advancing to term 238
I20240503 08:12:14.562009 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 237 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.562074 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 237 FOLLOWER]: Advancing to term 238
I20240503 08:12:14.562791 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.562791 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 238 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.562942 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 237. Current term is 238. Ops: []
I20240503 08:12:14.562973 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 238 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.562978 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 238 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.563176 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 238 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.563221 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 238 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.563287 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 238 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 238: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:14.563325 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 237.14346, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.563347 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 238 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.563345 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 238 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.563412 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 238: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:14.563544 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 238 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.563683 26604 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Leader election lost for term 238. Reason: could not achieve majority
I20240503 08:12:14.563714 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 238 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.563804 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 238 FOLLOWER]: Leader election lost for term 238. Reason: could not achieve majority
I20240503 08:12:14.568907 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 238 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 237. Current term is 238. Ops: []
I20240503 08:12:14.569195 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 237.14346, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.591477 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 237. Current term is 238. Ops: []
I20240503 08:12:14.591840 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 237.14346, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.594895 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.594972 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 238 FOLLOWER]: Advancing to term 239
I20240503 08:12:14.595491 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 238 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.595551 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 238 FOLLOWER]: Advancing to term 239
I20240503 08:12:14.595785 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 239 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.596043 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 239 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.596216 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 239 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.596249 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 239 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.596258 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 239 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.596401 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 239 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 239: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:14.596495 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 239 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.596518 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 239 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.596598 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 239 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.596660 26604 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 239 FOLLOWER]: Leader election lost for term 239. Reason: could not achieve majority
I20240503 08:12:14.596676 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 239 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.596735 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 239 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 239: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:14.596861 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 239 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.597038 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 239 FOLLOWER]: Leader election lost for term 239. Reason: could not achieve majority
I20240503 08:12:14.599370 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 239 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 237. Current term is 239. Ops: []
I20240503 08:12:14.599629 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 237.14346, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.618660 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 239 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 237. Current term is 239. Ops: []
I20240503 08:12:14.618963 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 237.14346, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.630666 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 239 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.630781 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 239 FOLLOWER]: Advancing to term 240
I20240503 08:12:14.631438 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 240 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.631595 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 240 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.631973 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 240 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.631909 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 240 candidate_status { last_received { term: 237 index: 14346 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.632129 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 239 FOLLOWER]: Advancing to term 240
I20240503 08:12:14.632798 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 240 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 240.
I20240503 08:12:14.633064 26334 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 237 LEADER]: Stepping down as leader of term 237
I20240503 08:12:14.633086 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 240 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:14.633143 26334 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 237 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:14.633260 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 240 FOLLOWER]: Leader election won for term 240
I20240503 08:12:14.633248 26334 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14346, Committed index: 14346, Last appended: 237.14346, Last appended by leader: 14346, Current term: 237, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.633345 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 240 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:14.633383 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 237 FOLLOWER]: Advancing to term 238
I20240503 08:12:14.633394 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14346, Committed index: 14346, Last appended: 237.14346, Last appended by leader: 14346, Current term: 240, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.634104 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 237 to 240, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 240 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:14.659492 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 240 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 237 index: 14346. Preceding OpId from leader: term: 240 index: 14347. (index mismatch)
I20240503 08:12:14.659915 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.662700 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 238 FOLLOWER]: Advancing to term 240
I20240503 08:12:14.663612 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 240 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 237 index: 14346. Preceding OpId from leader: term: 240 index: 14347. (index mismatch)
I20240503 08:12:14.663882 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14347, Last known committed idx: 14346, Time since last communication: 0.000s
I20240503 08:12:14.691800 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 240 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:14.691907 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 240 FOLLOWER]: Advancing to term 241
I20240503 08:12:14.692602 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 241 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.692744 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 241 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 240. Current term is 241. Ops: []
I20240503 08:12:14.692843 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 241 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.692911 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 241 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.692911 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 241 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.693002 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 240.14347, Next index: 14348, Last known committed idx: 14347, Time since last communication: 0.000s
I20240503 08:12:14.693137 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 241 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.693220 26604 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 241 FOLLOWER]: Leader election lost for term 241. Reason: could not achieve majority
I20240503 08:12:14.725109 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 241 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.725211 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 241 FOLLOWER]: Advancing to term 242
I20240503 08:12:14.725899 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.726068 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 242 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.726279 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 242 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.726279 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 242 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.726557 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 242 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:14.726689 26604 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Leader election lost for term 242. Reason: could not achieve majority
I20240503 08:12:14.727252 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 240. Current term is 242. Ops: []
I20240503 08:12:14.727460 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 240.14347, Next index: 14348, Last known committed idx: 14347, Time since last communication: 0.000s
I20240503 08:12:14.742103 26556 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 240 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:14.742201 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 240 FOLLOWER]: Advancing to term 241
I20240503 08:12:14.742950 26556 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 241 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.743100 26556 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 241 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.743341 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 241 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.743377 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 241 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.743443 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 241 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 240. Current term is 241. Ops: []
I20240503 08:12:14.743474 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 241. Current term is 242.
I20240503 08:12:14.743604 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 241 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 241. Current term is 242.
I20240503 08:12:14.743639 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 241 election: Cancelling election due to peer responding with higher term
I20240503 08:12:14.743644 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 240.14347, Next index: 14348, Last known committed idx: 14347, Time since last communication: 0.000s
I20240503 08:12:14.743717 26556 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 241 FOLLOWER]: Advancing to term 242
I20240503 08:12:14.744308 26556 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 242 FOLLOWER]: Leader election lost for term 241. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 241. Current term is 242.
I20240503 08:12:14.752405 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 240 LEADER]: Stepping down as leader of term 240
I20240503 08:12:14.752501 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 240 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:14.752578 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14347, Committed index: 14347, Last appended: 240.14347, Last appended by leader: 14347, Current term: 240, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.752703 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 240 FOLLOWER]: Advancing to term 241
I20240503 08:12:14.769073 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.769184 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 242 FOLLOWER]: Advancing to term 243
I20240503 08:12:14.769871 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 243 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.770043 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 243 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.770277 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 243 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.770277 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 243 candidate_status { last_received { term: 240 index: 14347 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.770387 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 242 FOLLOWER]: Advancing to term 243
I20240503 08:12:14.770396 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 241 FOLLOWER]: Advancing to term 243
I20240503 08:12:14.771099 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 243 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 243.
I20240503 08:12:14.771099 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 243 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 243.
I20240503 08:12:14.771329 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 243 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:14.771433 26604 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 243 FOLLOWER]: Leader election won for term 243
I20240503 08:12:14.771507 26604 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 243 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:14.771566 26604 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14347, Committed index: 14347, Last appended: 240.14347, Last appended by leader: 14347, Current term: 243, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.772037 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 240 to 243, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 243 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:14.801501 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 243 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 240 index: 14347. Preceding OpId from leader: term: 243 index: 14348. (index mismatch)
I20240503 08:12:14.801762 26604 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14348, Last known committed idx: 14347, Time since last communication: 0.000s
I20240503 08:12:14.802415 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 243 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 240 index: 14347. Preceding OpId from leader: term: 243 index: 14348. (index mismatch)
I20240503 08:12:14.802649 26604 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14348, Last known committed idx: 14347, Time since last communication: 0.000s
I20240503 08:12:14.833062 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 243 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:14.833179 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 243 FOLLOWER]: Advancing to term 244
I20240503 08:12:14.833978 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 244 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.834156 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 244 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.834103 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 244 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 243. Current term is 244. Ops: []
I20240503 08:12:14.834323 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 244 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.834324 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 244 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.834589 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 243.14348, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.834599 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 244 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.834735 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 244 FOLLOWER]: Leader election lost for term 244. Reason: could not achieve majority
I20240503 08:12:14.867007 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 244 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 243. Current term is 244. Ops: []
I20240503 08:12:14.867286 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 243.14348, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.871253 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 244 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.871317 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 244 FOLLOWER]: Advancing to term 245
I20240503 08:12:14.871948 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.872095 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 245 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.872318 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 245 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.872318 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 245 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.872561 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 245 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.872684 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Leader election lost for term 245. Reason: could not achieve majority
I20240503 08:12:14.881677 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 243 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:14.881755 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 243 FOLLOWER]: Advancing to term 244
I20240503 08:12:14.882440 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 244 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.882555 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 244 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 243. Current term is 244. Ops: []
I20240503 08:12:14.882592 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 244 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.882738 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 244 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.882824 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 243.14348, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.882910 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 244 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.882972 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 244. Current term is 245.
I20240503 08:12:14.883128 26173 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 244 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 244. Current term is 245.
I20240503 08:12:14.883175 26173 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 244 election: Cancelling election due to peer responding with higher term
I20240503 08:12:14.883256 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 244 FOLLOWER]: Advancing to term 245
I20240503 08:12:14.883807 26334 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 245 FOLLOWER]: Leader election lost for term 244. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 244. Current term is 245.
I20240503 08:12:14.894531 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 243. Current term is 245. Ops: []
I20240503 08:12:14.894785 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 243.14348, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.907974 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 245 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 243. Current term is 245. Ops: []
I20240503 08:12:14.908239 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 243.14348, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.916803 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 243. Current term is 245. Ops: []
I20240503 08:12:14.917109 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 243.14348, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.920428 26334 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 245 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:14.920495 26334 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 245 FOLLOWER]: Advancing to term 246
I20240503 08:12:14.921169 26334 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 246 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.921314 26334 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 246 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:14.921540 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 246 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:14.921540 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 246 candidate_status { last_received { term: 243 index: 14348 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.921656 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 245 FOLLOWER]: Advancing to term 246
I20240503 08:12:14.922233 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 246 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 246.
I20240503 08:12:14.922420 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 246 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.922509 26334 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 246 FOLLOWER]: Leader election won for term 246
I20240503 08:12:14.922573 26334 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 246 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:14.922649 26334 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14348, Committed index: 14348, Last appended: 243.14348, Last appended by leader: 14348, Current term: 246, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.923099 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 243 to 246, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 246 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:14.933421 26604 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 243 LEADER]: Stepping down as leader of term 243
I20240503 08:12:14.933531 26604 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 243 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:14.933595 26604 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14348, Committed index: 14348, Last appended: 243.14348, Last appended by leader: 14348, Current term: 243, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.933709 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 243 FOLLOWER]: Advancing to term 244
I20240503 08:12:14.947791 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 244 FOLLOWER]: Advancing to term 246
I20240503 08:12:14.947882 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 246 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 243 index: 14348. Preceding OpId from leader: term: 246 index: 14349. (index mismatch)
I20240503 08:12:14.948161 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.949045 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 246 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 243 index: 14348. Preceding OpId from leader: term: 246 index: 14349. (index mismatch)
I20240503 08:12:14.949246 26334 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14349, Last known committed idx: 14348, Time since last communication: 0.000s
I20240503 08:12:14.980406 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 246 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:14.980508 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 246 FOLLOWER]: Advancing to term 247
I20240503 08:12:14.981161 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:14.981336 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 247 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:14.981514 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 247 candidate_status { last_received { term: 246 index: 14349 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:14.981546 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 247 candidate_status { last_received { term: 246 index: 14349 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:14.981748 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 247 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:14.981845 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Leader election lost for term 247. Reason: could not achieve majority
I20240503 08:12:14.983889 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 246. Current term is 247. Ops: []
I20240503 08:12:14.984127 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 246.14349, Next index: 14350, Last known committed idx: 14349, Time since last communication: 0.000s
I20240503 08:12:15.008491 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 246 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:15.008605 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 246 FOLLOWER]: Advancing to term 247
I20240503 08:12:15.009442 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 247 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.009606 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 247 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.009800 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 247 candidate_status { last_received { term: 246 index: 14349 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.009868 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 247: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:15.009819 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 247 candidate_status { last_received { term: 246 index: 14349 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.010037 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 247 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.010140 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 247 FOLLOWER]: Leader election lost for term 247. Reason: could not achieve majority
I20240503 08:12:15.012723 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 247 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 246. Current term is 247. Ops: []
I20240503 08:12:15.012987 26556 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 246.14349, Next index: 14350, Last known committed idx: 14349, Time since last communication: 0.000s
I20240503 08:12:15.014950 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 246. Current term is 247. Ops: []
I20240503 08:12:15.015182 26334 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 246.14349, Next index: 14350, Last known committed idx: 14349, Time since last communication: 0.000s
I20240503 08:12:15.020610 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.020697 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 247 FOLLOWER]: Advancing to term 248
I20240503 08:12:15.021332 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 248 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.021469 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 248 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.021667 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 248 candidate_status { last_received { term: 246 index: 14349 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.021677 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 248 candidate_status { last_received { term: 246 index: 14349 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.021744 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 247 FOLLOWER]: Advancing to term 248
I20240503 08:12:15.022279 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 248 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 248.
I20240503 08:12:15.022522 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 248 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:15.022701 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 248 FOLLOWER]: Leader election won for term 248
I20240503 08:12:15.022761 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 248 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.022862 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14349, Committed index: 14349, Last appended: 246.14349, Last appended by leader: 14349, Current term: 248, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.023325 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 246 to 248, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 248 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:15.032814 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 246 LEADER]: Stepping down as leader of term 246
I20240503 08:12:15.032913 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 246 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:15.032999 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14349, Committed index: 14349, Last appended: 246.14349, Last appended by leader: 14349, Current term: 246, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.033149 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 246 FOLLOWER]: Advancing to term 247
I20240503 08:12:15.045756 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 248 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 246 index: 14349. Preceding OpId from leader: term: 248 index: 14350. (index mismatch)
I20240503 08:12:15.046104 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14350, Last known committed idx: 14349, Time since last communication: 0.000s
I20240503 08:12:15.050937 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 247 FOLLOWER]: Advancing to term 248
I20240503 08:12:15.051704 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 248 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 246 index: 14349. Preceding OpId from leader: term: 248 index: 14350. (index mismatch)
I20240503 08:12:15.052032 26567 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14350, Last known committed idx: 14349, Time since last communication: 0.000s
W20240503 08:12:15.055125 26348 lock_manager.cc:494] Txn 8 has not acquired the partition lock after 1000ms
I20240503 08:12:15.078016 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 248 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.078094 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 248 FOLLOWER]: Advancing to term 249
I20240503 08:12:15.078768 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.078936 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 249 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.079152 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 249 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.079162 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 249 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.079475 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 249 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.079569 26604 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Leader election lost for term 249. Reason: could not achieve majority
I20240503 08:12:15.082029 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 248. Current term is 249. Ops: []
I20240503 08:12:15.082312 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 248.14350, Next index: 14351, Last known committed idx: 14350, Time since last communication: 0.000s
I20240503 08:12:15.109479 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 248 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.109593 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 248 FOLLOWER]: Advancing to term 249
I20240503 08:12:15.110340 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 249 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.110518 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 249 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.110754 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 249 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.110831 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 249: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:15.110808 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 249 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.111078 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 249 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.111202 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 249 FOLLOWER]: Leader election lost for term 249. Reason: could not achieve majority
I20240503 08:12:15.116429 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 249 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 248. Current term is 249. Ops: []
I20240503 08:12:15.116740 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 248.14350, Next index: 14351, Last known committed idx: 14350, Time since last communication: 0.000s
I20240503 08:12:15.118572 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 248. Current term is 249. Ops: []
I20240503 08:12:15.118856 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 248.14350, Next index: 14351, Last known committed idx: 14350, Time since last communication: 0.000s
I20240503 08:12:15.121292 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.121342 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 249 FOLLOWER]: Advancing to term 250
I20240503 08:12:15.121980 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 250 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.122121 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 250 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.122270 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 250 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.122270 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 250 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.122387 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 249 FOLLOWER]: Advancing to term 250
I20240503 08:12:15.122997 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 250 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 250.
I20240503 08:12:15.123174 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 250 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.123262 26604 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 250 FOLLOWER]: Leader election won for term 250
I20240503 08:12:15.123324 26604 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 250 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:15.123383 26604 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14350, Committed index: 14350, Last appended: 248.14350, Last appended by leader: 14350, Current term: 250, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.123854 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 248 to 250, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 250 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:15.125578 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 248 LEADER]: Stepping down as leader of term 248
I20240503 08:12:15.125631 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 248 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.125698 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14350, Committed index: 14350, Last appended: 248.14350, Last appended by leader: 14350, Current term: 248, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.125803 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 248 FOLLOWER]: Advancing to term 249
I20240503 08:12:15.148087 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 249 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.148196 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 249 FOLLOWER]: Advancing to term 250
I20240503 08:12:15.148936 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 250 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.149096 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 250 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.149299 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 250 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.149317 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 250 candidate_status { last_received { term: 248 index: 14350 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.149443 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 250 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 250: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:15.149502 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 250 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 248 index: 14350. Preceding OpId from leader: term: 250 index: 14351. (index mismatch)
I20240503 08:12:15.149637 26604 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14351, Last known committed idx: 14350, Time since last communication: 0.000s
I20240503 08:12:15.149658 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 250 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.149780 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 250 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 248 index: 14350. Preceding OpId from leader: term: 250 index: 14351. (index mismatch)
I20240503 08:12:15.149827 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 250 FOLLOWER]: Leader election lost for term 250. Reason: could not achieve majority
I20240503 08:12:15.149987 26604 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14351, Last known committed idx: 14350, Time since last communication: 0.000s
I20240503 08:12:15.181571 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 250 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:15.181667 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 250 FOLLOWER]: Advancing to term 251
I20240503 08:12:15.182336 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 251 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.182518 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 251 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.182736 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 251 candidate_status { last_received { term: 250 index: 14351 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.182735 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 251 candidate_status { last_received { term: 250 index: 14351 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.182927 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 251 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 250. Current term is 251. Ops: []
I20240503 08:12:15.183076 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 251 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.183176 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 251 FOLLOWER]: Leader election lost for term 251. Reason: could not achieve majority
I20240503 08:12:15.183193 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 250.14351, Next index: 14352, Last known committed idx: 14351, Time since last communication: 0.000s
I20240503 08:12:15.208094 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 251 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 250. Current term is 251. Ops: []
I20240503 08:12:15.208372 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 250.14351, Next index: 14352, Last known committed idx: 14351, Time since last communication: 0.000s
I20240503 08:12:15.214946 26604 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 250 LEADER]: Stepping down as leader of term 250
I20240503 08:12:15.215044 26604 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 250 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:15.215123 26604 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14351, Committed index: 14351, Last appended: 250.14351, Last appended by leader: 14351, Current term: 250, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.215240 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 250 FOLLOWER]: Advancing to term 251
I20240503 08:12:15.225939 26539 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 251 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.226035 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 251 FOLLOWER]: Advancing to term 252
I20240503 08:12:15.226706 26539 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 252 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.226900 26539 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 252 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.227088 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 252 candidate_status { last_received { term: 250 index: 14351 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.227113 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 252 candidate_status { last_received { term: 250 index: 14351 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.227212 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 251 FOLLOWER]: Advancing to term 252
I20240503 08:12:15.227844 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 252.
I20240503 08:12:15.228036 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 252 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:15.228133 26539 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 252 FOLLOWER]: Leader election won for term 252
I20240503 08:12:15.228186 26539 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 252 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.228243 26539 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14351, Committed index: 14351, Last appended: 250.14351, Last appended by leader: 14351, Current term: 252, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.228706 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 250 to 252, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 252 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:15.231194 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 250 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:15.231241 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 250 FOLLOWER]: Advancing to term 251
I20240503 08:12:15.231788 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 251 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.231922 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 251 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.232132 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 251 candidate_status { last_received { term: 250 index: 14351 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.232156 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 251 candidate_status { last_received { term: 250 index: 14351 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.232270 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 251. Current term is 252.
I20240503 08:12:15.232434 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 251 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 251. Current term is 252.
I20240503 08:12:15.232468 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 251 election: Cancelling election due to peer responding with higher term
I20240503 08:12:15.232517 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 251 FOLLOWER]: Advancing to term 252
I20240503 08:12:15.233007 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 252 FOLLOWER]: Leader election lost for term 251. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 251. Current term is 252.
I20240503 08:12:15.251111 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 250 index: 14351. Preceding OpId from leader: term: 252 index: 14352. (index mismatch)
I20240503 08:12:15.251390 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14352, Last known committed idx: 14351, Time since last communication: 0.000s
I20240503 08:12:15.251847 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 252 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 250 index: 14351. Preceding OpId from leader: term: 252 index: 14352. (index mismatch)
I20240503 08:12:15.252005 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14352, Last known committed idx: 14351, Time since last communication: 0.000s
I20240503 08:12:15.283048 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.283154 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 252 FOLLOWER]: Advancing to term 253
I20240503 08:12:15.283843 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.284013 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 253 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.284205 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 253 candidate_status { last_received { term: 252 index: 14352 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.284221 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 253 candidate_status { last_received { term: 252 index: 14352 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.284456 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 253 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.284539 26604 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Leader election lost for term 253. Reason: could not achieve majority
I20240503 08:12:15.286939 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 252. Current term is 253. Ops: []
I20240503 08:12:15.287180 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 252.14352, Next index: 14353, Last known committed idx: 14352, Time since last communication: 0.000s
I20240503 08:12:15.308080 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 252 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.308182 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 252 FOLLOWER]: Advancing to term 253
I20240503 08:12:15.308868 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 253 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.309053 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 253 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.309296 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 253 candidate_status { last_received { term: 252 index: 14352 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.309295 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 253 candidate_status { last_received { term: 252 index: 14352 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.309417 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 253: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:15.309577 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 253 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.309679 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 253 FOLLOWER]: Leader election lost for term 253. Reason: could not achieve majority
I20240503 08:12:15.310225 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 252. Current term is 253. Ops: []
I20240503 08:12:15.310369 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 253 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 252. Current term is 253. Ops: []
I20240503 08:12:15.310407 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 252.14352, Next index: 14353, Last known committed idx: 14352, Time since last communication: 0.000s
I20240503 08:12:15.310576 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 252.14352, Next index: 14353, Last known committed idx: 14352, Time since last communication: 0.000s
I20240503 08:12:15.316287 26604 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.316356 26604 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 253 FOLLOWER]: Advancing to term 254
I20240503 08:12:15.316982 26604 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 254 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.317121 26604 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 254 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.317324 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 254 candidate_status { last_received { term: 252 index: 14352 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.317324 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 254 candidate_status { last_received { term: 252 index: 14352 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.317466 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 253 FOLLOWER]: Advancing to term 254
I20240503 08:12:15.318099 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 254 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 254.
I20240503 08:12:15.318339 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 254 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.318439 26604 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 254 FOLLOWER]: Leader election won for term 254
I20240503 08:12:15.318509 26604 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 254 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:15.318560 26604 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14352, Committed index: 14352, Last appended: 252.14352, Last appended by leader: 14352, Current term: 254, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.319056 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 252 to 254, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 254 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:15.337208 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 254 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 252. Current term is 254. Ops: []
I20240503 08:12:15.337495 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 252.14352, Next index: 14353, Last known committed idx: 14352, Time since last communication: 0.000s
I20240503 08:12:15.341589 26284 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 252 LEADER]: Stepping down as leader of term 252
I20240503 08:12:15.341672 26284 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 252 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.341744 26284 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14352, Committed index: 14352, Last appended: 252.14352, Last appended by leader: 14352, Current term: 252, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.341867 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 252 FOLLOWER]: Advancing to term 254
I20240503 08:12:15.342535 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 254 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 252 index: 14352. Preceding OpId from leader: term: 254 index: 14353. (index mismatch)
I20240503 08:12:15.342846 26604 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14353, Last known committed idx: 14352, Time since last communication: 0.000s
I20240503 08:12:15.355657 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 254 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 252 index: 14352. Preceding OpId from leader: term: 254 index: 14353. (index mismatch)
I20240503 08:12:15.355955 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14353, Last known committed idx: 14352, Time since last communication: 0.000s
I20240503 08:12:15.375346 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 254 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:15.375464 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 254 FOLLOWER]: Advancing to term 255
I20240503 08:12:15.376118 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 255 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.376308 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 255 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.376515 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 255 candidate_status { last_received { term: 254 index: 14353 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.376515 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 255 candidate_status { last_received { term: 254 index: 14353 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.376747 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 255 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.376874 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 255 FOLLOWER]: Leader election lost for term 255. Reason: could not achieve majority
I20240503 08:12:15.379160 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 255 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 254. Current term is 255. Ops: []
I20240503 08:12:15.379405 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 254.14353, Next index: 14354, Last known committed idx: 14353, Time since last communication: 0.000s
I20240503 08:12:15.410560 26603 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 254 LEADER]: Stepping down as leader of term 254
I20240503 08:12:15.410674 26603 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 254 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:15.410755 26603 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14353, Committed index: 14353, Last appended: 254.14353, Last appended by leader: 14353, Current term: 254, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.410880 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 254 FOLLOWER]: Advancing to term 255
I20240503 08:12:15.414041 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 255 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.414126 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 255 FOLLOWER]: Advancing to term 256
I20240503 08:12:15.414791 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 256 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.415048 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 256 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.415174 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 256 candidate_status { last_received { term: 254 index: 14353 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.415217 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 256 candidate_status { last_received { term: 254 index: 14353 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.415293 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 255 FOLLOWER]: Advancing to term 256
I20240503 08:12:15.415947 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 256.
I20240503 08:12:15.416116 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 256 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:15.416213 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 256 FOLLOWER]: Leader election won for term 256
I20240503 08:12:15.416267 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 256 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.416308 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14353, Committed index: 14353, Last appended: 254.14353, Last appended by leader: 14353, Current term: 256, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.416755 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 254 to 256, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 256 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:15.431234 26539 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 254 current term: 256 is higher.
I20240503 08:12:15.438519 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 254 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:15.438637 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 254 FOLLOWER]: Advancing to term 255
I20240503 08:12:15.439352 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 255 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.439510 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 255 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.439749 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 255 candidate_status { last_received { term: 254 index: 14353 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.439747 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 255 candidate_status { last_received { term: 254 index: 14353 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.439877 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 255. Current term is 256.
I20240503 08:12:15.440052 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 255 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 255. Current term is 256.
I20240503 08:12:15.440121 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 255 election: Cancelling election due to peer responding with higher term
I20240503 08:12:15.440233 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 255 FOLLOWER]: Advancing to term 256
I20240503 08:12:15.440763 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 256 FOLLOWER]: Leader election lost for term 255. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 255. Current term is 256.
I20240503 08:12:15.441388 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 256 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 254 index: 14353. Preceding OpId from leader: term: 256 index: 14354. (index mismatch)
I20240503 08:12:15.441565 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14354, Last known committed idx: 14353, Time since last communication: 0.000s
I20240503 08:12:15.442600 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 254 index: 14353. Preceding OpId from leader: term: 256 index: 14354. (index mismatch)
I20240503 08:12:15.442800 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14354, Last known committed idx: 14353, Time since last communication: 0.000s
I20240503 08:12:15.499981 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.500080 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 256 FOLLOWER]: Advancing to term 257
I20240503 08:12:15.500739 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 257 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.500905 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 257 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.501047 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 257 candidate_status { last_received { term: 256 index: 14354 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.501109 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 257 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 256. Current term is 257. Ops: []
I20240503 08:12:15.501082 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 257 candidate_status { last_received { term: 256 index: 14354 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.501288 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 257 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.501339 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 256.14354, Next index: 14355, Last known committed idx: 14354, Time since last communication: 0.000s
I20240503 08:12:15.501384 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 257 FOLLOWER]: Leader election lost for term 257. Reason: could not achieve majority
I20240503 08:12:15.502630 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 256 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.502679 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 256 FOLLOWER]: Advancing to term 257
I20240503 08:12:15.503237 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 257 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.503434 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 257 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.503528 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 257 candidate_status { last_received { term: 256 index: 14354 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.503530 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 257 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 256. Current term is 257. Ops: []
I20240503 08:12:15.503631 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 257 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 257: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:15.503556 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 257 candidate_status { last_received { term: 256 index: 14354 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.503787 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 256.14354, Next index: 14355, Last known committed idx: 14354, Time since last communication: 0.000s
I20240503 08:12:15.503865 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 257 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.503937 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 257 FOLLOWER]: Leader election lost for term 257. Reason: could not achieve majority
I20240503 08:12:15.506527 26539 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 256 LEADER]: Stepping down as leader of term 256
I20240503 08:12:15.506557 26539 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 256 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.506599 26539 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14354, Committed index: 14354, Last appended: 256.14354, Last appended by leader: 14354, Current term: 256, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.506713 26539 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 256 FOLLOWER]: Advancing to term 257
I20240503 08:12:15.531031 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 257 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.531144 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 257 FOLLOWER]: Advancing to term 258
I20240503 08:12:15.531808 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 258 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.531982 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 258 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.532179 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 258 candidate_status { last_received { term: 256 index: 14354 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.532245 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 257 FOLLOWER]: Advancing to term 258
I20240503 08:12:15.532184 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 258 candidate_status { last_received { term: 256 index: 14354 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.532296 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 257 FOLLOWER]: Advancing to term 258
I20240503 08:12:15.532780 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 258 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 258.
I20240503 08:12:15.532994 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 258 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:15.533041 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 258 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 258.
I20240503 08:12:15.533111 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 258 FOLLOWER]: Leader election won for term 258
I20240503 08:12:15.533175 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 258 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.533222 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14354, Committed index: 14354, Last appended: 256.14354, Last appended by leader: 14354, Current term: 258, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.533715 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 256 to 258. New cstate: current_term: 258 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:15.546494 26539 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 257 current term: 258 is higher.
I20240503 08:12:15.557164 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 258 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 256 index: 14354. Preceding OpId from leader: term: 258 index: 14355. (index mismatch)
I20240503 08:12:15.557456 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14355, Last known committed idx: 14354, Time since last communication: 0.000s
I20240503 08:12:15.563480 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 258 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 256 index: 14354. Preceding OpId from leader: term: 258 index: 14355. (index mismatch)
I20240503 08:12:15.563767 26539 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14355, Last known committed idx: 14354, Time since last communication: 0.000s
I20240503 08:12:15.588726 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 258 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.588836 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 258 FOLLOWER]: Advancing to term 259
I20240503 08:12:15.589537 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.589708 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 259 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.589919 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 259 candidate_status { last_received { term: 258 index: 14355 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.589919 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 259 candidate_status { last_received { term: 258 index: 14355 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.590188 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 259 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.590319 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Leader election lost for term 259. Reason: could not achieve majority
I20240503 08:12:15.594355 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 258 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.594425 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 258 FOLLOWER]: Advancing to term 259
I20240503 08:12:15.595130 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 258. Current term is 259. Ops: []
I20240503 08:12:15.595216 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 259 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.595394 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 259 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 258. Current term is 259. Ops: []
I20240503 08:12:15.595346 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 258.14355, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.595404 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 259 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.595530 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 259 candidate_status { last_received { term: 258 index: 14355 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.595553 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 259 candidate_status { last_received { term: 258 index: 14355 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.595571 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 258.14355, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.595629 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 259: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:15.595748 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 259 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.595840 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 259 FOLLOWER]: Leader election lost for term 259. Reason: could not achieve majority
I20240503 08:12:15.619513 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 259 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 258. Current term is 259. Ops: []
I20240503 08:12:15.619661 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 258. Current term is 259. Ops: []
I20240503 08:12:15.619797 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 258.14355, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.619891 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 258.14355, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.623081 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.623163 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 259 FOLLOWER]: Advancing to term 260
I20240503 08:12:15.623860 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.624007 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 260 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.624207 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 260 candidate_status { last_received { term: 258 index: 14355 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.624234 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 260 candidate_status { last_received { term: 258 index: 14355 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.624300 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 259 FOLLOWER]: Advancing to term 260
I20240503 08:12:15.624938 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 260 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 260.
I20240503 08:12:15.625104 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 260 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.625191 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 FOLLOWER]: Leader election won for term 260
I20240503 08:12:15.625254 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:15.625309 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14355, Committed index: 14355, Last appended: 258.14355, Last appended by leader: 14355, Current term: 260, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.625666 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 258 to 260, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 260 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:15.652395 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 260 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 258. Current term is 260. Ops: []
I20240503 08:12:15.652561 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 258. Current term is 260. Ops: []
I20240503 08:12:15.652676 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 258.14355, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.652787 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 258.14355, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.652886 26539 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 260.14356 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14355
I20240503 08:12:15.657203 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 260 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 258 index: 14355. Preceding OpId from leader: term: 260 index: 14356. (index mismatch)
I20240503 08:12:15.657467 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.659277 26284 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 258 LEADER]: Stepping down as leader of term 258
I20240503 08:12:15.659327 26284 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 258 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.659397 26284 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14355, Committed index: 14355, Last appended: 258.14355, Last appended by leader: 14355, Current term: 258, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.659484 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 258 FOLLOWER]: Advancing to term 260
I20240503 08:12:15.660080 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 260 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 258 index: 14355. Preceding OpId from leader: term: 260 index: 14356. (index mismatch)
I20240503 08:12:15.660351 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14356, Last known committed idx: 14355, Time since last communication: 0.000s
I20240503 08:12:15.689239 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 260 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:15.689333 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 260 FOLLOWER]: Advancing to term 261
I20240503 08:12:15.689981 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.690135 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 261 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.690272 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 261 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.690279 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 261 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.690518 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 261 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.690801 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Leader election lost for term 261. Reason: could not achieve majority
I20240503 08:12:15.694113 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 260. Current term is 261. Ops: []
I20240503 08:12:15.694347 26604 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 260.14356, Next index: 14357, Last known committed idx: 14356, Time since last communication: 0.000s
I20240503 08:12:15.719537 26567 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 260 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:15.719635 26567 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 260 FOLLOWER]: Advancing to term 261
I20240503 08:12:15.719888 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 260. Current term is 261. Ops: []
I20240503 08:12:15.720131 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 260.14356, Next index: 14357, Last known committed idx: 14356, Time since last communication: 0.000s
I20240503 08:12:15.720510 26567 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 261 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.720660 26567 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 261 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.720850 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 261 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 260. Current term is 261. Ops: []
I20240503 08:12:15.720829 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 261 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.720847 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 261 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.721009 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 261: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:15.721071 26604 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 260.14356, Next index: 14357, Last known committed idx: 14356, Time since last communication: 0.000s
I20240503 08:12:15.721293 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 261 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.721380 26567 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 261 FOLLOWER]: Leader election lost for term 261. Reason: could not achieve majority
I20240503 08:12:15.732332 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.732419 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 261 FOLLOWER]: Advancing to term 262
I20240503 08:12:15.733052 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 262 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.733209 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 262 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.733422 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 262 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.733420 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 262 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.733525 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 261 FOLLOWER]: Advancing to term 262
I20240503 08:12:15.734056 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 262 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 262.
I20240503 08:12:15.734263 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 262 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.734354 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 262 FOLLOWER]: Leader election won for term 262
I20240503 08:12:15.734449 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 262 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:15.734493 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14356, Committed index: 14356, Last appended: 260.14356, Last appended by leader: 14356, Current term: 262, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.734978 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 260 to 262, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 262 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:15.735776 26603 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 LEADER]: Stepping down as leader of term 260
I20240503 08:12:15.735813 26603 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:15.735865 26603 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14356, Committed index: 14356, Last appended: 260.14356, Last appended by leader: 14356, Current term: 260, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.735946 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 260 FOLLOWER]: Advancing to term 261
I20240503 08:12:15.760704 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 261 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.760813 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 261 FOLLOWER]: Advancing to term 262
I20240503 08:12:15.761624 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 262 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.761765 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 262 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 260 index: 14356. Preceding OpId from leader: term: 262 index: 14357. (index mismatch)
I20240503 08:12:15.761783 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 262 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.761966 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 262 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.762028 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14357, Last known committed idx: 14356, Time since last communication: 0.000s
I20240503 08:12:15.762060 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 262 candidate_status { last_received { term: 260 index: 14356 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.762174 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 262 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 262: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:15.762334 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 262 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.762415 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 262 FOLLOWER]: Leader election lost for term 262. Reason: could not achieve majority
I20240503 08:12:15.769582 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 262 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 260 index: 14356. Preceding OpId from leader: term: 262 index: 14357. (index mismatch)
I20240503 08:12:15.769850 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14357, Last known committed idx: 14356, Time since last communication: 0.000s
I20240503 08:12:15.801069 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 262 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:15.801190 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 262 FOLLOWER]: Advancing to term 263
I20240503 08:12:15.801847 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 263 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.801991 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 263 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.802109 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 263 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.802131 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 263 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.802367 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 263 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.802462 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 263 FOLLOWER]: Leader election lost for term 263. Reason: could not achieve majority
I20240503 08:12:15.835053 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 263 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.835150 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 263 FOLLOWER]: Advancing to term 264
I20240503 08:12:15.835811 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.835975 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 264 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.836200 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 264 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.836189 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 264 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.836395 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 262. Current term is 264. Ops: []
I20240503 08:12:15.836549 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 264 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:15.836563 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 262.14357, Next index: 14358, Last known committed idx: 14357, Time since last communication: 0.000s
I20240503 08:12:15.836669 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Leader election lost for term 264. Reason: could not achieve majority
I20240503 08:12:15.859949 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 262. Current term is 264. Ops: []
I20240503 08:12:15.860209 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 262.14357, Next index: 14358, Last known committed idx: 14357, Time since last communication: 0.000s
I20240503 08:12:15.872982 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 262 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:15.873082 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 262 FOLLOWER]: Advancing to term 263
I20240503 08:12:15.873787 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 263 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.873952 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 263 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.874153 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 263 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.874150 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 263 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.874285 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 263. Current term is 264.
I20240503 08:12:15.874406 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 263 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 262. Current term is 263. Ops: []
I20240503 08:12:15.874440 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 263 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 263. Current term is 264.
I20240503 08:12:15.874480 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 263 election: Cancelling election due to peer responding with higher term
I20240503 08:12:15.874562 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 263 FOLLOWER]: Advancing to term 264
I20240503 08:12:15.874693 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 262.14357, Next index: 14358, Last known committed idx: 14357, Time since last communication: 0.000s
I20240503 08:12:15.875193 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 264 FOLLOWER]: Leader election lost for term 263. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 263. Current term is 264.
I20240503 08:12:15.879142 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.879207 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 264 FOLLOWER]: Advancing to term 265
I20240503 08:12:15.879740 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 265 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.879899 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 265 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.880084 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 265 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.880084 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 265 candidate_status { last_received { term: 262 index: 14357 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.880187 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 264 FOLLOWER]: Advancing to term 265
I20240503 08:12:15.880932 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 265 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 265.
I20240503 08:12:15.881129 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 265 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:15.881220 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 265 FOLLOWER]: Leader election won for term 265
I20240503 08:12:15.881275 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 265 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.881322 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14357, Committed index: 14357, Last appended: 262.14357, Last appended by leader: 14357, Current term: 265, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.881767 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 262 to 265, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 265 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:15.884347 26611 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 262 LEADER]: Stepping down as leader of term 262
I20240503 08:12:15.884397 26611 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 262 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:15.884474 26611 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14357, Committed index: 14357, Last appended: 262.14357, Last appended by leader: 14357, Current term: 262, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.884582 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 262 FOLLOWER]: Advancing to term 264
I20240503 08:12:15.904134 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 264 FOLLOWER]: Advancing to term 265
I20240503 08:12:15.904834 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 265 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 262 index: 14357. Preceding OpId from leader: term: 265 index: 14358. (index mismatch)
I20240503 08:12:15.905107 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14358, Last known committed idx: 14357, Time since last communication: 0.000s
I20240503 08:12:15.915669 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 265 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 262 index: 14357. Preceding OpId from leader: term: 265 index: 14358. (index mismatch)
I20240503 08:12:15.915979 26517 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14358, Last known committed idx: 14357, Time since last communication: 0.000s
I20240503 08:12:15.936076 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 265 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.936173 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 265 FOLLOWER]: Advancing to term 266
I20240503 08:12:15.936897 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 266 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.937069 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 266 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.937325 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 266 candidate_status { last_received { term: 265 index: 14358 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.937340 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 266 candidate_status { last_received { term: 265 index: 14358 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.937587 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 266 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.937687 26611 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 266 FOLLOWER]: Leader election lost for term 266. Reason: could not achieve majority
I20240503 08:12:15.942600 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 266 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 265. Current term is 266. Ops: []
I20240503 08:12:15.943090 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 265.14358, Next index: 14359, Last known committed idx: 14358, Time since last communication: 0.000s
I20240503 08:12:15.970427 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 265 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:15.970547 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 265 FOLLOWER]: Advancing to term 266
I20240503 08:12:15.971256 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 266 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.971418 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 266 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:15.971619 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 266 candidate_status { last_received { term: 265 index: 14358 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.971750 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 266 candidate_status { last_received { term: 265 index: 14358 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:15.971813 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 266 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 266: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:15.971979 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 266 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.972066 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 266 FOLLOWER]: Leader election lost for term 266. Reason: could not achieve majority
I20240503 08:12:15.974411 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 266 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:15.974462 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 266 FOLLOWER]: Advancing to term 267
I20240503 08:12:15.975010 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 266 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 265. Current term is 266. Ops: []
I20240503 08:12:15.975195 26567 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 265.14358, Next index: 14359, Last known committed idx: 14358, Time since last communication: 0.000s
I20240503 08:12:15.975293 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.975409 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 265. Current term is 267. Ops: []
I20240503 08:12:15.975498 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 267 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:15.975551 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 267 candidate_status { last_received { term: 265 index: 14358 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:15.975667 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 267 candidate_status { last_received { term: 265 index: 14358 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:15.975649 26539 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 265.14358, Next index: 14359, Last known committed idx: 14358, Time since last communication: 0.000s
I20240503 08:12:15.975759 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 266 FOLLOWER]: Advancing to term 267
I20240503 08:12:15.976279 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 267 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 267.
I20240503 08:12:15.976440 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 267 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:15.976517 26611 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 FOLLOWER]: Leader election won for term 267
I20240503 08:12:15.976570 26611 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:15.976608 26611 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14358, Committed index: 14358, Last appended: 265.14358, Last appended by leader: 14358, Current term: 267, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.977020 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 265 to 267, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 267 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:15.995412 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 265 LEADER]: Stepping down as leader of term 265
I20240503 08:12:15.995501 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 265 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:15.995564 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14358, Committed index: 14358, Last appended: 265.14358, Last appended by leader: 14358, Current term: 265, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:15.995680 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 265 FOLLOWER]: Advancing to term 266
I20240503 08:12:15.999199 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 266 FOLLOWER]: Advancing to term 267
I20240503 08:12:15.999804 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 267 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 265 index: 14358. Preceding OpId from leader: term: 267 index: 14359. (index mismatch)
I20240503 08:12:16.000033 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14359, Last known committed idx: 14358, Time since last communication: 0.000s
I20240503 08:12:16.011500 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 267 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 265 index: 14358. Preceding OpId from leader: term: 267 index: 14359. (index mismatch)
I20240503 08:12:16.011777 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14359, Last known committed idx: 14358, Time since last communication: 0.000s
I20240503 08:12:16.042479 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 267 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.042591 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 267 FOLLOWER]: Advancing to term 268
I20240503 08:12:16.043355 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 268 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.043537 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 268 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.043772 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 268 candidate_status { last_received { term: 267 index: 14359 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.043778 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 268 candidate_status { last_received { term: 267 index: 14359 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.044018 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 268 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.044111 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 268 FOLLOWER]: Leader election lost for term 268. Reason: could not achieve majority
I20240503 08:12:16.048067 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 268 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 267. Current term is 268. Ops: []
I20240503 08:12:16.048310 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 267.14359, Next index: 14360, Last known committed idx: 14359, Time since last communication: 0.000s
W20240503 08:12:16.056208 26351 lock_manager.cc:494] Txn 8 has not acquired the partition lock after 1000ms
I20240503 08:12:16.057231 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 268 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 267. Current term is 268. Ops: [267.14360-267.14360]
I20240503 08:12:16.057551 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 267.14359, Next index: 14360, Last known committed idx: 14359, Time since last communication: 0.000s
I20240503 08:12:16.082648 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 268 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.082751 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 268 FOLLOWER]: Advancing to term 269
I20240503 08:12:16.083488 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.083665 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 269 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.083925 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 269 candidate_status { last_received { term: 267 index: 14359 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.083905 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 269 candidate_status { last_received { term: 267 index: 14359 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.084208 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 269 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.084326 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Leader election lost for term 269. Reason: could not achieve majority
I20240503 08:12:16.087320 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 267. Current term is 269. Ops: [267.14360-267.14360]
I20240503 08:12:16.087579 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 267.14359, Next index: 14360, Last known committed idx: 14359, Time since last communication: 0.000s
I20240503 08:12:16.097265 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 LEADER]: Stepping down as leader of term 267
I20240503 08:12:16.097357 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.097448 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14360, Committed index: 14360, Last appended: 267.14360, Last appended by leader: 14360, Current term: 267, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.097568 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 267 FOLLOWER]: Advancing to term 268
I20240503 08:12:16.117655 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 267 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.117797 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 267 FOLLOWER]: Advancing to term 268
I20240503 08:12:16.118753 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 268 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.119100 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 268 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.119336 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 268 candidate_status { last_received { term: 267 index: 14360 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.119316 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 268 candidate_status { last_received { term: 267 index: 14360 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.119460 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 268. Current term is 269.
I20240503 08:12:16.119655 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 268 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 268. Current term is 269.
I20240503 08:12:16.119688 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 268 election: Cancelling election due to peer responding with higher term
I20240503 08:12:16.119773 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 268 FOLLOWER]: Advancing to term 269
I20240503 08:12:16.120380 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 268 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 268.
I20240503 08:12:16.120548 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 269 FOLLOWER]: Leader election lost for term 268. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 268. Current term is 269.
I20240503 08:12:16.142407 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.142520 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 269 FOLLOWER]: Advancing to term 270
I20240503 08:12:16.143390 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 270 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.143581 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 270 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.143776 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 270 candidate_status { last_received { term: 267 index: 14359 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.143843 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 269 FOLLOWER]: Advancing to term 270
I20240503 08:12:16.143779 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 270 candidate_status { last_received { term: 267 index: 14359 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.143879 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 268 FOLLOWER]: Advancing to term 270
I20240503 08:12:16.144433 26284 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 270 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 270 because replica has last-logged OpId of term: 267 index: 14360, which is greater than that of the candidate, which has last-logged OpId of term: 267 index: 14359.
I20240503 08:12:16.144749 26209 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 270 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for term 270 because replica has last-logged OpId of term: 267 index: 14360, which is greater than that of the candidate, which has last-logged OpId of term: 267 index: 14359.
I20240503 08:12:16.144953 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 270 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.145067 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 270 FOLLOWER]: Leader election lost for term 270. Reason: could not achieve majority
I20240503 08:12:16.151464 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 270 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.151584 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 270 FOLLOWER]: Advancing to term 271
I20240503 08:12:16.152423 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 271 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.152616 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 271 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.152837 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 271 candidate_status { last_received { term: 267 index: 14360 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.152911 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 270 FOLLOWER]: Advancing to term 271
I20240503 08:12:16.152915 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 271 candidate_status { last_received { term: 267 index: 14360 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.152981 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 270 FOLLOWER]: Advancing to term 271
I20240503 08:12:16.153725 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 271 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 271.
I20240503 08:12:16.153725 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 271 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 271.
I20240503 08:12:16.153959 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 271 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:16.154071 26611 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 271 FOLLOWER]: Leader election won for term 271
I20240503 08:12:16.154114 26611 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 271 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.154168 26611 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14359, Committed index: 14359, Last appended: 267.14360, Last appended by leader: 14360, Current term: 271, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.154671 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 267 to 271. New cstate: current_term: 271 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:16.172082 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 271 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 267 index: 14360. Preceding OpId from leader: term: 271 index: 14361. (index mismatch)
I20240503 08:12:16.172089 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 271 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 267 index: 14359. Preceding OpId from leader: term: 271 index: 14361. (index mismatch)
I20240503 08:12:16.172456 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14361, Last known committed idx: 14360, Time since last communication: 0.000s
I20240503 08:12:16.172552 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14361, Last known committed idx: 14359, Time since last communication: 0.000s
W20240503 08:12:16.181885 26611 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 269 current term: 271 is higher.
I20240503 08:12:16.203868 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 271 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.203969 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 271 FOLLOWER]: Advancing to term 272
I20240503 08:12:16.204465 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 271 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.204527 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 271 FOLLOWER]: Advancing to term 272
I20240503 08:12:16.204715 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 272 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.204883 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 272 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.205099 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 272 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.205104 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 272 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.205242 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 272 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.205377 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 272 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 272: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:16.205421 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 272 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.205535 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 272 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.205539 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 272 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.205657 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 272 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 272: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:16.205694 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 272 FOLLOWER]: Leader election lost for term 272. Reason: could not achieve majority
I20240503 08:12:16.205727 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 272 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.205932 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 272 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:16.206032 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 272 FOLLOWER]: Leader election lost for term 272. Reason: could not achieve majority
I20240503 08:12:16.207340 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 272 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 271. Current term is 272. Ops: []
I20240503 08:12:16.207538 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 271.14361, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.208250 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 272 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 271. Current term is 272. Ops: []
I20240503 08:12:16.208434 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 271.14361, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.232736 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 271 LEADER]: Stepping down as leader of term 271
I20240503 08:12:16.232837 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 271 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.232923 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14361, Committed index: 14361, Last appended: 271.14361, Last appended by leader: 14361, Current term: 271, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.233048 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 271 FOLLOWER]: Advancing to term 272
I20240503 08:12:16.247879 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 272 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.247984 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 272 FOLLOWER]: Advancing to term 273
I20240503 08:12:16.248767 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 273 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.248939 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 273 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.249195 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 273 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.249207 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 273 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.249264 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 272 FOLLOWER]: Advancing to term 273
I20240503 08:12:16.249286 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 272 FOLLOWER]: Advancing to term 273
I20240503 08:12:16.249868 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 273 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 273.
I20240503 08:12:16.249933 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 273 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 273.
I20240503 08:12:16.250118 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 273 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:16.250218 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 273 FOLLOWER]: Leader election won for term 273
I20240503 08:12:16.250300 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 273 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.250366 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14361, Committed index: 14361, Last appended: 271.14361, Last appended by leader: 14361, Current term: 273, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.250942 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 271 to 273, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 273 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.280647 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 273 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.280735 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 273 FOLLOWER]: Advancing to term 274
I20240503 08:12:16.281380 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 274 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.281524 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 274 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.281701 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 273 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.281746 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 274 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.281812 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 273 FOLLOWER]: Advancing to term 274
I20240503 08:12:16.281713 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 274 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.282466 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.282584 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 274: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:16.282642 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 274 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.282780 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 274 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.282791 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 274 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:16.282796 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 274 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.282930 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 274 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 274: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:16.282980 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 274 FOLLOWER]: Leader election lost for term 274. Reason: could not achieve majority
I20240503 08:12:16.283121 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 274 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.283118 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 273. Current term is 274. Ops: []
I20240503 08:12:16.283118 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 274 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 273. Current term is 274. Ops: []
I20240503 08:12:16.283250 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Leader election lost for term 274. Reason: could not achieve majority
I20240503 08:12:16.283346 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.283506 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.317476 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 274 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 273. Current term is 274. Ops: [273.14362-273.14362]
I20240503 08:12:16.317859 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 271.14361, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.318361 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 273. Current term is 274. Ops: [273.14362-273.14362]
I20240503 08:12:16.318575 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 271.14361, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.321797 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.321866 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 274 FOLLOWER]: Advancing to term 275
I20240503 08:12:16.322556 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 275 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.322754 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 275 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.322899 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 275 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.322913 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 275 candidate_status { last_received { term: 271 index: 14361 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.322978 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 274 FOLLOWER]: Advancing to term 275
I20240503 08:12:16.323549 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 275 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 275.
I20240503 08:12:16.323737 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 275 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:16.323837 26612 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 275 FOLLOWER]: Leader election won for term 275
I20240503 08:12:16.323901 26612 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 275 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.323959 26612 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14361, Committed index: 14361, Last appended: 271.14361, Last appended by leader: 14361, Current term: 275, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.324399 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 273 to 275, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 275 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:16.324604 26515 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 273 LEADER]: Stepping down as leader of term 273
I20240503 08:12:16.324651 26515 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 273 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.324707 26515 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14361, Committed index: 14361, Last appended: 273.14362, Last appended by leader: 14362, Current term: 273, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.324795 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 273 FOLLOWER]: Advancing to term 274
I20240503 08:12:16.349397 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 274 FOLLOWER]: Advancing to term 275
I20240503 08:12:16.350199 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 275 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 273 index: 14362. Preceding OpId from leader: term: 275 index: 14362. (term mismatch)
I20240503 08:12:16.350244 26134 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting all ops after (but not including) 14361
I20240503 08:12:16.350258 26134 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Aborting uncommitted NO_OP operation due to leader change: 273.14362
I20240503 08:12:16.350275 26134 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 275 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:16.350663 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.358872 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 275 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 271 index: 14361. Preceding OpId from leader: term: 275 index: 14362. (index mismatch)
I20240503 08:12:16.359136 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14362, Last known committed idx: 14361, Time since last communication: 0.000s
I20240503 08:12:16.382664 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 275 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.382783 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 275 FOLLOWER]: Advancing to term 276
I20240503 08:12:16.383574 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 276 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.383734 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 276 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.383965 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 276 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.383980 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 276 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.384209 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 276 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.384357 26626 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 276 FOLLOWER]: Leader election lost for term 276. Reason: could not achieve majority
I20240503 08:12:16.388347 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 276 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 275. Current term is 276. Ops: []
I20240503 08:12:16.388653 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 275.14362, Next index: 14363, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.414848 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 276 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 275. Current term is 276. Ops: []
I20240503 08:12:16.415179 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 275.14362, Next index: 14363, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.417009 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 276 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.417071 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 276 FOLLOWER]: Advancing to term 277
I20240503 08:12:16.417834 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.417994 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 277 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.418154 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 277 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.418180 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 277 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.418421 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 277 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.418545 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Leader election lost for term 277. Reason: could not achieve majority
I20240503 08:12:16.427718 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 275 LEADER]: Stepping down as leader of term 275
I20240503 08:12:16.427814 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 275 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.427891 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14362, Committed index: 14362, Last appended: 275.14362, Last appended by leader: 14362, Current term: 275, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.428012 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 275 FOLLOWER]: Advancing to term 276
I20240503 08:12:16.444110 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 275 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.444223 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 275 FOLLOWER]: Advancing to term 276
I20240503 08:12:16.444922 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 276 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.445096 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 276 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.445313 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 276 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.445319 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 276 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.445430 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 276. Current term is 277.
I20240503 08:12:16.445647 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 276 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 276. Current term is 277.
I20240503 08:12:16.445710 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 276 election: Cancelling election due to peer responding with higher term
I20240503 08:12:16.445780 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 276 FOLLOWER]: Advancing to term 277
I20240503 08:12:16.446004 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 276 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 276.
I20240503 08:12:16.446357 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 277 FOLLOWER]: Leader election lost for term 276. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 276. Current term is 277.
I20240503 08:12:16.480473 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.480576 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 277 FOLLOWER]: Advancing to term 278
I20240503 08:12:16.481283 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 278 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.481451 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 278 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.481662 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 278 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.481683 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 278 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.481766 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 277 FOLLOWER]: Advancing to term 278
I20240503 08:12:16.481772 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 276 FOLLOWER]: Advancing to term 278
I20240503 08:12:16.482484 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 278 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 278.
I20240503 08:12:16.482484 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 278 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 278.
I20240503 08:12:16.482745 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 278 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:16.482880 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 278 FOLLOWER]: Leader election won for term 278
I20240503 08:12:16.482944 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 278 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.483009 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14362, Committed index: 14362, Last appended: 275.14362, Last appended by leader: 14362, Current term: 278, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.483522 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 275 to 278, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 278 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.505685 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 278 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 275 index: 14362. Preceding OpId from leader: term: 278 index: 14363. (index mismatch)
I20240503 08:12:16.505960 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14363, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.513496 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 278 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.513576 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 278 FOLLOWER]: Advancing to term 279
I20240503 08:12:16.514216 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.514356 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 279 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.514539 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 279 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.514547 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 279 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.514825 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 279 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:16.514930 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Leader election lost for term 279. Reason: could not achieve majority
I20240503 08:12:16.518944 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 278. Current term is 279. Ops: []
I20240503 08:12:16.519155 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14363, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.531065 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 278. Current term is 279. Ops: [278.14363-278.14363]
I20240503 08:12:16.531325 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 275.14362, Next index: 14363, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.537257 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 278 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:16.537335 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 278 FOLLOWER]: Advancing to term 279
I20240503 08:12:16.538004 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 279 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.538146 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 279 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.538332 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 279 candidate_status { last_received { term: 278 index: 14363 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.538332 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 279 candidate_status { last_received { term: 278 index: 14363 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.538450 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 279: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:16.538592 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 279 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.538699 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 279 FOLLOWER]: Leader election lost for term 279. Reason: could not achieve majority
I20240503 08:12:16.540380 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 279 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 278. Current term is 279. Ops: []
I20240503 08:12:16.540612 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 278.14363, Next index: 14364, Last known committed idx: 14363, Time since last communication: 0.000s
I20240503 08:12:16.562894 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.562991 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 279 FOLLOWER]: Advancing to term 280
I20240503 08:12:16.563702 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 280 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.563946 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 280 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.564034 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 280 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.564034 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 280 candidate_status { last_received { term: 275 index: 14362 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.564152 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 279 FOLLOWER]: Advancing to term 280
I20240503 08:12:16.564751 26209 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 280 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for term 280 because replica has last-logged OpId of term: 278 index: 14363, which is greater than that of the candidate, which has last-logged OpId of term: 275 index: 14362.
I20240503 08:12:16.564956 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 280 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:16.565053 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 280 FOLLOWER]: Leader election lost for term 280. Reason: could not achieve majority
I20240503 08:12:16.566548 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 280 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 278. Current term is 280. Ops: [278.14363-278.14363]
I20240503 08:12:16.566763 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 275.14362, Next index: 14363, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.568737 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 280 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 278. Current term is 280. Ops: []
I20240503 08:12:16.568981 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 278.14363, Next index: 14364, Last known committed idx: 14363, Time since last communication: 0.000s
I20240503 08:12:16.570123 26603 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 278 LEADER]: Stepping down as leader of term 278
I20240503 08:12:16.570171 26603 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 278 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.570219 26603 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14363, Committed index: 14363, Last appended: 278.14363, Last appended by leader: 14363, Current term: 278, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.570348 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 278 FOLLOWER]: Advancing to term 279
I20240503 08:12:16.580421 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 280 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.580508 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 280 FOLLOWER]: Advancing to term 281
I20240503 08:12:16.581151 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 281 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.581334 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 281 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.581533 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 281 candidate_status { last_received { term: 278 index: 14363 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.581542 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 281 candidate_status { last_received { term: 278 index: 14363 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.581604 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 280 FOLLOWER]: Advancing to term 281
I20240503 08:12:16.581650 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 279 FOLLOWER]: Advancing to term 281
I20240503 08:12:16.582180 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 281 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 281.
I20240503 08:12:16.582192 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 281 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 281.
I20240503 08:12:16.582386 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 281 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:16.582484 26612 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 281 FOLLOWER]: Leader election won for term 281
I20240503 08:12:16.582548 26612 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 281 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.582582 26612 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14363, Committed index: 14363, Last appended: 278.14363, Last appended by leader: 14363, Current term: 281, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.583134 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 278 to 281, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 281 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:16.617714 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 281 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 275 index: 14362. Preceding OpId from leader: term: 281 index: 14364. (index mismatch)
I20240503 08:12:16.618039 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14364, Last known committed idx: 14362, Time since last communication: 0.000s
I20240503 08:12:16.619159 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 281 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 278 index: 14363. Preceding OpId from leader: term: 281 index: 14364. (index mismatch)
I20240503 08:12:16.619369 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14364, Last known committed idx: 14363, Time since last communication: 0.000s
I20240503 08:12:16.649801 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 281 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.649901 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 281 FOLLOWER]: Advancing to term 282
I20240503 08:12:16.650540 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 282 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.650683 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 282 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 281. Current term is 282. Ops: []
I20240503 08:12:16.650801 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 282 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.650893 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 282 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.650913 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 281.14364, Next index: 14365, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.650887 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 282 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.651224 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 282 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.651337 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 282 FOLLOWER]: Leader election lost for term 282. Reason: could not achieve majority
I20240503 08:12:16.672181 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 282 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 281. Current term is 282. Ops: []
I20240503 08:12:16.672497 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 281.14364, Next index: 14365, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.688859 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 282 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.688975 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 282 FOLLOWER]: Advancing to term 283
I20240503 08:12:16.689671 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.689839 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 283 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.690052 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 283 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.690060 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 283 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.690400 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 283 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.690532 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Leader election lost for term 283. Reason: could not achieve majority
I20240503 08:12:16.697870 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 281 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:16.697963 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 281 FOLLOWER]: Advancing to term 282
I20240503 08:12:16.698762 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 281. Current term is 283. Ops: []
I20240503 08:12:16.698727 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 282 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.698922 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 282 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.699049 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 282 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.699136 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 281.14364, Next index: 14365, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.699237 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 282 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.699316 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 282. Current term is 283.
I20240503 08:12:16.699559 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 282 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 282. Current term is 283.
I20240503 08:12:16.699607 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 282 election: Cancelling election due to peer responding with higher term
I20240503 08:12:16.699674 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 282 FOLLOWER]: Advancing to term 283
I20240503 08:12:16.700191 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 283 FOLLOWER]: Leader election lost for term 282. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 282. Current term is 283.
I20240503 08:12:16.700701 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 283 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 281. Current term is 283. Ops: []
I20240503 08:12:16.700938 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 281.14364, Next index: 14365, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.718654 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 281 LEADER]: Stepping down as leader of term 281
I20240503 08:12:16.718776 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 281 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.718858 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14364, Committed index: 14364, Last appended: 281.14364, Last appended by leader: 14364, Current term: 281, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.719003 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 281 FOLLOWER]: Advancing to term 282
I20240503 08:12:16.722154 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.722234 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 283 FOLLOWER]: Advancing to term 284
I20240503 08:12:16.722891 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 284 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.723063 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 284 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.723284 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 284 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.723294 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 284 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.723392 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 283 FOLLOWER]: Advancing to term 284
I20240503 08:12:16.723395 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 282 FOLLOWER]: Advancing to term 284
I20240503 08:12:16.724063 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 284 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 284.
I20240503 08:12:16.724063 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 284 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 284.
I20240503 08:12:16.724323 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 284 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:16.724427 26515 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 284 FOLLOWER]: Leader election won for term 284
I20240503 08:12:16.724478 26515 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 284 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.724526 26515 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14364, Committed index: 14364, Last appended: 281.14364, Last appended by leader: 14364, Current term: 284, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.724980 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 281 to 284, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 284 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.754638 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 284 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.754750 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 284 FOLLOWER]: Advancing to term 285
I20240503 08:12:16.755666 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 285 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.755851 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 285 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.755802 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 285 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 284. Current term is 285. Ops: []
I20240503 08:12:16.755996 26285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 285 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.756047 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 285 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.756069 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14365, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.755769 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 284 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 281 index: 14364. Preceding OpId from leader: term: 284 index: 14365. (index mismatch)
I20240503 08:12:16.756222 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 285 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.756389 26611 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 285 FOLLOWER]: Leader election lost for term 285. Reason: could not achieve majority
I20240503 08:12:16.756486 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14365, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.777315 26603 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 284 LEADER]: Stepping down as leader of term 284
I20240503 08:12:16.777415 26603 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 284 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.777495 26603 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14365, Committed index: 14365, Last appended: 284.14365, Last appended by leader: 14365, Current term: 284, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.777647 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 284 FOLLOWER]: Advancing to term 285
I20240503 08:12:16.788487 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 284 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:16.788623 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 284 FOLLOWER]: Advancing to term 285
I20240503 08:12:16.789367 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 285 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.789614 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 285 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.789791 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 285 candidate_status { last_received { term: 284 index: 14365 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.789851 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 285 candidate_status { last_received { term: 284 index: 14365 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.789927 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 285 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 285: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:16.790515 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 285 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 285.
I20240503 08:12:16.790719 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 285 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:16.790839 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 285 FOLLOWER]: Leader election won for term 285
I20240503 08:12:16.790972 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 285 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:16.791033 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14365, Committed index: 14365, Last appended: 284.14365, Last appended by leader: 14365, Current term: 285, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.791615 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 284 to 285, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 285 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.793092 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 285 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.793136 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 285 FOLLOWER]: Advancing to term 286
I20240503 08:12:16.793669 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 286 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.793787 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 286 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.793953 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 286 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.793952 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 286 candidate_status { last_received { term: 281 index: 14364 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.794044 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 285 FOLLOWER]: Advancing to term 286
I20240503 08:12:16.794636 26134 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 286 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 286 because replica has last-logged OpId of term: 284 index: 14365, which is greater than that of the candidate, which has last-logged OpId of term: 281 index: 14364.
I20240503 08:12:16.794816 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 286 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.794929 26611 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 286 FOLLOWER]: Leader election lost for term 286. Reason: could not achieve majority
I20240503 08:12:16.820001 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 286 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 285. Current term is 286. Ops: []
I20240503 08:12:16.820315 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14366, Last known committed idx: 14365, Time since last communication: 0.000s
I20240503 08:12:16.821161 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 286 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.821215 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 286 FOLLOWER]: Advancing to term 287
I20240503 08:12:16.821897 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 287 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.822038 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 287 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.822181 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 287 candidate_status { last_received { term: 284 index: 14365 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.822201 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 287 candidate_status { last_received { term: 284 index: 14365 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.822261 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 286 FOLLOWER]: Advancing to term 287
I20240503 08:12:16.822886 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 287 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 287.
I20240503 08:12:16.823087 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 287 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.823175 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 287 FOLLOWER]: Leader election won for term 287
I20240503 08:12:16.823244 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 287 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.823304 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14364, Committed index: 14364, Last appended: 284.14365, Last appended by leader: 14365, Current term: 287, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.823798 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 285 to 287, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 287 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:16.825135 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 287 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 285. Current term is 287. Ops: []
I20240503 08:12:16.825343 26627 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14366, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.828548 26517 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 285 LEADER]: Stepping down as leader of term 285
I20240503 08:12:16.828599 26517 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 285 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:16.828652 26517 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14365, Committed index: 14365, Last appended: 285.14366, Last appended by leader: 14366, Current term: 285, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.828733 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 285 FOLLOWER]: Advancing to term 286
I20240503 08:12:16.847004 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 286 FOLLOWER]: Advancing to term 287
I20240503 08:12:16.847719 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 287 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 285 index: 14366. Preceding OpId from leader: term: 287 index: 14366. (term mismatch)
I20240503 08:12:16.847761 26284 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14365
I20240503 08:12:16.847782 26284 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted NO_OP operation due to leader change: 285.14366
I20240503 08:12:16.847805 26284 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 287 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:16.848037 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14366, Last known committed idx: 14365, Time since last communication: 0.000s
I20240503 08:12:16.852207 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 287 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 281 index: 14364. Preceding OpId from leader: term: 287 index: 14366. (index mismatch)
I20240503 08:12:16.852483 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14366, Last known committed idx: 14364, Time since last communication: 0.000s
I20240503 08:12:16.935946 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 287 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:16.936050 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 287 FOLLOWER]: Advancing to term 288
I20240503 08:12:16.936760 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 288 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.936928 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 288 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.937136 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 288 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.937145 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 288 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.937386 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 288 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:16.937477 26611 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 288 FOLLOWER]: Leader election lost for term 288. Reason: could not achieve majority
I20240503 08:12:16.942028 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 288 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 287. Current term is 288. Ops: []
I20240503 08:12:16.942385 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 287.14366, Next index: 14367, Last known committed idx: 14366, Time since last communication: 0.000s
I20240503 08:12:16.953248 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 287 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:16.953346 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 287 FOLLOWER]: Advancing to term 288
I20240503 08:12:16.954020 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 288 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.954200 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 288 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:16.954418 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 288 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.954439 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 288 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:16.954527 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 288 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 288: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:16.954691 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 288 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:16.954808 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 288 FOLLOWER]: Leader election lost for term 288. Reason: could not achieve majority
I20240503 08:12:16.957476 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 288 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 287. Current term is 288. Ops: []
I20240503 08:12:16.957705 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 287.14366, Next index: 14367, Last known committed idx: 14366, Time since last communication: 0.000s
I20240503 08:12:16.958581 26603 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 287 LEADER]: Stepping down as leader of term 287
I20240503 08:12:16.958640 26603 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 287 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:16.958693 26603 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14366, Committed index: 14366, Last appended: 287.14366, Last appended by leader: 14366, Current term: 287, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.958775 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 287 FOLLOWER]: Advancing to term 288
I20240503 08:12:16.980331 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 288 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:16.980437 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 288 FOLLOWER]: Advancing to term 289
I20240503 08:12:16.981133 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 289 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.981307 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 289 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:16.981598 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 289 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:16.981598 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 289 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:16.981724 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 288 FOLLOWER]: Advancing to term 289
I20240503 08:12:16.981724 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 288 FOLLOWER]: Advancing to term 289
I20240503 08:12:16.982339 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 289 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 289.
I20240503 08:12:16.982362 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 289 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 289.
I20240503 08:12:16.982564 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 289 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:16.982692 26611 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 289 FOLLOWER]: Leader election won for term 289
I20240503 08:12:16.982753 26611 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 289 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:16.982801 26611 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14366, Committed index: 14366, Last appended: 287.14366, Last appended by leader: 14366, Current term: 289, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:16.983265 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 287 to 289, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 289 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:17.012838 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 289 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.012952 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 289 FOLLOWER]: Advancing to term 290
I20240503 08:12:17.013613 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.013778 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 290 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.014008 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 290 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.014068 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 289 FOLLOWER]: Advancing to term 290
I20240503 08:12:17.014039 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 290 candidate_status { last_received { term: 287 index: 14366 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.014732 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 290 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 290.
I20240503 08:12:17.014977 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 290 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:17.015132 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 FOLLOWER]: Leader election won for term 290
I20240503 08:12:17.015223 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:17.015303 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14366, Committed index: 14366, Last appended: 287.14366, Last appended by leader: 14366, Current term: 290, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.015834 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 289 to 290, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 290 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.017762 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 289. Current term is 290. Ops: []
I20240503 08:12:17.018010 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14367, Last known committed idx: 14366, Time since last communication: 0.000s
I20240503 08:12:17.018096 26611 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 290.14367 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14366
I20240503 08:12:17.019649 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 290 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 289. Current term is 290. Ops: []
I20240503 08:12:17.019850 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14367, Last known committed idx: 14366, Time since last communication: 0.000s
I20240503 08:12:17.024219 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 289 LEADER]: Stepping down as leader of term 289
I20240503 08:12:17.024281 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 289 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.024359 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14366, Committed index: 14366, Last appended: 289.14367, Last appended by leader: 14367, Current term: 289, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.024459 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 289 FOLLOWER]: Advancing to term 290
I20240503 08:12:17.040836 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 290 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 287 index: 14366. Preceding OpId from leader: term: 290 index: 14367. (index mismatch)
I20240503 08:12:17.041090 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14367, Last known committed idx: 14366, Time since last communication: 0.000s
I20240503 08:12:17.045635 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 290 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 289 index: 14367. Preceding OpId from leader: term: 290 index: 14367. (term mismatch)
I20240503 08:12:17.045708 26209 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting all ops after (but not including) 14366
I20240503 08:12:17.045722 26209 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Aborting uncommitted NO_OP operation due to leader change: 289.14367
I20240503 08:12:17.045737 26209 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 290 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:17.046010 26515 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14367, Last known committed idx: 14366, Time since last communication: 0.000s
I20240503 08:12:17.077047 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 290 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:17.077148 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 290 FOLLOWER]: Advancing to term 291
I20240503 08:12:17.077865 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.078040 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 291 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.078037 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 290. Current term is 291. Ops: []
I20240503 08:12:17.078217 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 291 candidate_status { last_received { term: 290 index: 14367 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.078233 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 291 candidate_status { last_received { term: 290 index: 14367 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.078464 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 290.14367, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.078491 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 291 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.078634 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Leader election lost for term 291. Reason: could not achieve majority
I20240503 08:12:17.098121 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 290 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:17.098228 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 290 FOLLOWER]: Advancing to term 291
I20240503 08:12:17.098942 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 291 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.099201 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 291 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.099334 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 291 candidate_status { last_received { term: 290 index: 14367 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.099339 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 291 candidate_status { last_received { term: 290 index: 14367 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.099448 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 291: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:17.099715 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 291 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.099828 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 291 FOLLOWER]: Leader election lost for term 291. Reason: could not achieve majority
I20240503 08:12:17.101948 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 291 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 290. Current term is 291. Ops: []
I20240503 08:12:17.102160 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 290.14367, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.102308 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 290. Current term is 291. Ops: []
I20240503 08:12:17.102604 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 290.14367, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.123199 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.123312 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 291 FOLLOWER]: Advancing to term 292
I20240503 08:12:17.123983 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.124181 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 292 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.124390 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 292 candidate_status { last_received { term: 290 index: 14367 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.124390 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 292 candidate_status { last_received { term: 290 index: 14367 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.124508 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 291 FOLLOWER]: Advancing to term 292
I20240503 08:12:17.125133 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 292 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 292.
I20240503 08:12:17.125353 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 292 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.125500 26612 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 FOLLOWER]: Leader election won for term 292
I20240503 08:12:17.125576 26612 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.125644 26612 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14367, Committed index: 14367, Last appended: 290.14367, Last appended by leader: 14367, Current term: 292, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.126045 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 290. Current term is 292. Ops: []
I20240503 08:12:17.126082 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 290 to 292, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 292 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:17.126260 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 290.14367, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.126350 26603 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 292.14368 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14367
I20240503 08:12:17.132159 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 292 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 290. Current term is 292. Ops: []
I20240503 08:12:17.132467 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 290.14367, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.151226 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 290. Current term is 292. Ops: []
I20240503 08:12:17.151572 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 290.14367, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.151665 26603 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 292.14368 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14367
I20240503 08:12:17.154428 26626 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 LEADER]: Stepping down as leader of term 290
I20240503 08:12:17.154475 26626 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:17.154523 26626 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14367, Committed index: 14367, Last appended: 290.14367, Last appended by leader: 14367, Current term: 290, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.154667 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 290 FOLLOWER]: Advancing to term 291
I20240503 08:12:17.158483 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 292 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 290 index: 14367. Preceding OpId from leader: term: 292 index: 14368. (index mismatch)
I20240503 08:12:17.158735 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
I20240503 08:12:17.162268 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 291 FOLLOWER]: Advancing to term 292
I20240503 08:12:17.162916 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 292 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 290 index: 14367. Preceding OpId from leader: term: 292 index: 14368. (index mismatch)
I20240503 08:12:17.163141 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14368, Last known committed idx: 14367, Time since last communication: 0.000s
W20240503 08:12:17.172469 26348 lock_manager.cc:494] Txn 9 has not acquired the partition lock after 1000ms
I20240503 08:12:17.193993 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 292 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.194094 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 292 FOLLOWER]: Advancing to term 293
I20240503 08:12:17.194793 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 293 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.194962 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 293 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.195139 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 293 candidate_status { last_received { term: 292 index: 14368 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.195139 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 293 candidate_status { last_received { term: 292 index: 14368 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.195436 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 293 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.195551 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 293 FOLLOWER]: Leader election lost for term 293. Reason: could not achieve majority
I20240503 08:12:17.195909 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 293 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 292. Current term is 293. Ops: []
I20240503 08:12:17.196094 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 292.14368, Next index: 14369, Last known committed idx: 14368, Time since last communication: 0.000s
I20240503 08:12:17.209296 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 LEADER]: Stepping down as leader of term 292
I20240503 08:12:17.209386 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.209460 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14368, Committed index: 14368, Last appended: 292.14368, Last appended by leader: 14368, Current term: 292, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.209578 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 292 FOLLOWER]: Advancing to term 293
I20240503 08:12:17.214826 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 292 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.214923 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 292 FOLLOWER]: Advancing to term 293
I20240503 08:12:17.215561 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 293 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.215732 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 293 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.215960 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 293 candidate_status { last_received { term: 292 index: 14368 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.215960 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 293 candidate_status { last_received { term: 292 index: 14368 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.216085 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 293 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 293: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:17.216655 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 293 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 293.
I20240503 08:12:17.216897 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 293 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.216982 26517 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 293 FOLLOWER]: Leader election won for term 293
I20240503 08:12:17.217049 26517 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 293 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:17.217104 26517 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14368, Committed index: 14368, Last appended: 292.14368, Last appended by leader: 14368, Current term: 293, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.217561 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 292 to 293, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 293 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.239619 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 293 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.239782 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 293 FOLLOWER]: Advancing to term 294
I20240503 08:12:17.240883 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.241099 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 294 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.241438 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 294 candidate_status { last_received { term: 292 index: 14368 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.241438 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 294 candidate_status { last_received { term: 292 index: 14368 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.241542 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 293 FOLLOWER]: Advancing to term 294
I20240503 08:12:17.242415 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 294 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 294.
I20240503 08:12:17.242537 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 294 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 293. Current term is 294. Ops: []
I20240503 08:12:17.242715 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 294 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.242815 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 FOLLOWER]: Leader election won for term 294
I20240503 08:12:17.242807 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14369, Last known committed idx: 14368, Time since last communication: 0.000s
I20240503 08:12:17.242928 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:17.242978 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14368, Committed index: 14368, Last appended: 292.14368, Last appended by leader: 14368, Current term: 294, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.246412 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 293 to 294, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 294 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.253340 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 LEADER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 293. Current term is 294. Ops: []
I20240503 08:12:17.253595 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14369, Last known committed idx: 14368, Time since last communication: 0.000s
I20240503 08:12:17.253684 26517 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 294.14369 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14368
I20240503 08:12:17.266716 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 294 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 293. Current term is 294. Ops: [293.14369-293.14369]
I20240503 08:12:17.266952 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 292.14368, Next index: 14369, Last known committed idx: 14368, Time since last communication: 0.000s
I20240503 08:12:17.268038 26627 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 293 LEADER]: Stepping down as leader of term 293
I20240503 08:12:17.268088 26627 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 293 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:17.268150 26627 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14368, Committed index: 14368, Last appended: 293.14369, Last appended by leader: 14369, Current term: 293, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.268263 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 293 FOLLOWER]: Advancing to term 294
I20240503 08:12:17.270819 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 294 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 293 index: 14369. Preceding OpId from leader: term: 294 index: 14369. (term mismatch)
I20240503 08:12:17.270869 26284 pending_rounds.cc:76] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting all ops after (but not including) 14368
I20240503 08:12:17.270884 26284 pending_rounds.cc:97] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Aborting uncommitted NO_OP operation due to leader change: 293.14369
I20240503 08:12:17.270900 26284 raft_consensus.cc:2866] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 294 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:17.271135 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14369, Last known committed idx: 14368, Time since last communication: 0.000s
I20240503 08:12:17.272697 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 294 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 292 index: 14368. Preceding OpId from leader: term: 294 index: 14369. (index mismatch)
I20240503 08:12:17.272913 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14369, Last known committed idx: 14368, Time since last communication: 0.000s
W20240503 08:12:17.282703 26626 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 291 current term: 294 is higher.
W20240503 08:12:17.320899 26626 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 292 current term: 294 is higher.
I20240503 08:12:17.330394 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 294 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:17.330495 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 294 FOLLOWER]: Advancing to term 295
I20240503 08:12:17.331229 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.331395 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 295 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.331544 26517 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 294 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:17.331602 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 295 candidate_status { last_received { term: 294 index: 14369 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.331609 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 295 candidate_status { last_received { term: 294 index: 14369 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.331624 26517 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 294 FOLLOWER]: Advancing to term 295
I20240503 08:12:17.332284 26517 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 295 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.332410 26517 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 295 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.332417 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 295 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 295: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:17.332564 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 295 candidate_status { last_received { term: 294 index: 14369 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.332633 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 295: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:17.332674 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 295 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.332649 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 295 candidate_status { last_received { term: 294 index: 14369 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.332816 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Leader election lost for term 295. Reason: could not achieve majority
I20240503 08:12:17.332926 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 295 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.333015 26517 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 295 FOLLOWER]: Leader election lost for term 295. Reason: could not achieve majority
I20240503 08:12:17.333880 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 294. Current term is 295. Ops: []
I20240503 08:12:17.334153 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 294.14369, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.337703 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 295 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 294. Current term is 295. Ops: []
I20240503 08:12:17.338032 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 294.14369, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.356230 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 294. Current term is 295. Ops: []
I20240503 08:12:17.356536 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 294.14369, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.361124 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 295 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 294. Current term is 295. Ops: []
I20240503 08:12:17.361374 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 294.14369, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.364482 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.364545 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 295 FOLLOWER]: Advancing to term 296
I20240503 08:12:17.365196 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.365335 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 296 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.365516 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 296 candidate_status { last_received { term: 294 index: 14369 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.365522 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 296 candidate_status { last_received { term: 294 index: 14369 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.365597 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 295 FOLLOWER]: Advancing to term 296
I20240503 08:12:17.366183 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 296 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 296.
I20240503 08:12:17.366355 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 296 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.366473 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 FOLLOWER]: Leader election won for term 296
I20240503 08:12:17.366529 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.366591 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14369, Committed index: 14369, Last appended: 294.14369, Last appended by leader: 14369, Current term: 296, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.367081 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 294 to 296, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 296 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:17.381255 26626 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 292 current term: 294 is higher.
I20240503 08:12:17.386766 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 294. Current term is 296. Ops: []
I20240503 08:12:17.387053 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 294.14369, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.387162 26603 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 296.14370 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14369
I20240503 08:12:17.392724 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 296 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 294 index: 14369. Preceding OpId from leader: term: 296 index: 14370. (index mismatch)
I20240503 08:12:17.393059 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.396406 26134 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 LEADER]: Stepping down as leader of term 294
I20240503 08:12:17.396468 26134 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:17.396528 26134 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14369, Committed index: 14369, Last appended: 294.14369, Last appended by leader: 14369, Current term: 294, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.396638 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 294 FOLLOWER]: Advancing to term 296
I20240503 08:12:17.397294 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 296 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 294 index: 14369. Preceding OpId from leader: term: 296 index: 14370. (index mismatch)
I20240503 08:12:17.397585 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14370, Last known committed idx: 14369, Time since last communication: 0.000s
I20240503 08:12:17.428292 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 296 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.428390 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 296 FOLLOWER]: Advancing to term 297
I20240503 08:12:17.429075 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 297 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.429333 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 297 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.429436 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 297 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.429455 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 297 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.429698 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 297 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.429790 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 297 FOLLOWER]: Leader election lost for term 297. Reason: could not achieve majority
I20240503 08:12:17.434989 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 297 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 296. Current term is 297. Ops: []
I20240503 08:12:17.435248 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 296.14370, Next index: 14371, Last known committed idx: 14370, Time since last communication: 0.000s
I20240503 08:12:17.469841 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 297 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 296. Current term is 297. Ops: []
I20240503 08:12:17.470144 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 296.14370, Next index: 14371, Last known committed idx: 14370, Time since last communication: 0.000s
I20240503 08:12:17.474287 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 297 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.474355 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 297 FOLLOWER]: Advancing to term 298
I20240503 08:12:17.475069 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.475306 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 298 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.475396 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 298 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.475401 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 298 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.475656 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 298 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.475754 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Leader election lost for term 298. Reason: could not achieve majority
I20240503 08:12:17.477767 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 296 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.477823 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 296 FOLLOWER]: Advancing to term 297
I20240503 08:12:17.478444 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 297 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.478603 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 297 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.478792 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 297 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.478792 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 297 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.478896 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 297. Current term is 298.
I20240503 08:12:17.478972 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 297 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 296. Current term is 297. Ops: []
I20240503 08:12:17.479128 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 297 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 297. Current term is 298.
I20240503 08:12:17.479166 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 297 election: Cancelling election due to peer responding with higher term
I20240503 08:12:17.479166 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 296.14370, Next index: 14371, Last known committed idx: 14370, Time since last communication: 0.000s
I20240503 08:12:17.479315 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 297 FOLLOWER]: Advancing to term 298
I20240503 08:12:17.479801 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 298 FOLLOWER]: Leader election lost for term 297. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 297. Current term is 298.
I20240503 08:12:17.480310 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 LEADER]: Stepping down as leader of term 296
I20240503 08:12:17.480348 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.480391 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14370, Committed index: 14370, Last appended: 296.14370, Last appended by leader: 14370, Current term: 296, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.480459 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 296 FOLLOWER]: Advancing to term 297
I20240503 08:12:17.502712 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 297 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.502812 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 297 FOLLOWER]: Advancing to term 298
I20240503 08:12:17.503453 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 298 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.503607 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 298 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.503831 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 298 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.503836 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 298 candidate_status { last_received { term: 296 index: 14370 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.503962 26134 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 298: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:17.504473 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 298 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 298.
I20240503 08:12:17.504664 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 298 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.504770 26611 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 298 FOLLOWER]: Leader election won for term 298
I20240503 08:12:17.504830 26611 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 298 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.504894 26611 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14370, Committed index: 14370, Last appended: 296.14370, Last appended by leader: 14370, Current term: 298, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.505393 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 296 to 298. New cstate: current_term: 298 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:17.527166 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 297 current term: 298 is higher.
I20240503 08:12:17.531827 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 296 index: 14370. Preceding OpId from leader: term: 298 index: 14371. (index mismatch)
I20240503 08:12:17.532097 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14371, Last known committed idx: 14370, Time since last communication: 0.000s
I20240503 08:12:17.534888 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 298 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 296 index: 14370. Preceding OpId from leader: term: 298 index: 14371. (index mismatch)
I20240503 08:12:17.535091 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14371, Last known committed idx: 14370, Time since last communication: 0.000s
W20240503 08:12:17.541386 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 297 current term: 298 is higher.
I20240503 08:12:17.564153 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.564251 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 298 FOLLOWER]: Advancing to term 299
I20240503 08:12:17.564940 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.565083 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 299 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.565306 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 299 candidate_status { last_received { term: 298 index: 14371 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.565362 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 298 FOLLOWER]: Advancing to term 299
I20240503 08:12:17.565371 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 299 candidate_status { last_received { term: 298 index: 14371 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.565578 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 298. Current term is 299. Ops: []
I20240503 08:12:17.565776 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 298.14371, Next index: 14372, Last known committed idx: 14371, Time since last communication: 0.000s
I20240503 08:12:17.566089 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 299 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 299.
I20240503 08:12:17.566272 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 299 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:17.566358 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 FOLLOWER]: Leader election won for term 299
I20240503 08:12:17.566419 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:17.566459 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14371, Committed index: 14371, Last appended: 298.14371, Last appended by leader: 14371, Current term: 299, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.566972 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 298 to 299, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 299 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.571866 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 299 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 298. Current term is 299. Ops: []
I20240503 08:12:17.572171 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 298.14371, Next index: 14372, Last known committed idx: 14371, Time since last communication: 0.000s
I20240503 08:12:17.593411 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 298. Current term is 299. Ops: []
I20240503 08:12:17.593703 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 298.14371, Next index: 14372, Last known committed idx: 14371, Time since last communication: 0.000s
I20240503 08:12:17.593791 26611 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer db1737abb1d5409bb049fa1c94d3ed48 log is divergent from this leader: its last log entry 299.14372 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14371
I20240503 08:12:17.597362 26210 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 298 LEADER]: Stepping down as leader of term 298
I20240503 08:12:17.597419 26210 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 298 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.597483 26210 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14371, Committed index: 14371, Last appended: 298.14371, Last appended by leader: 14371, Current term: 298, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.597594 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 298 FOLLOWER]: Advancing to term 299
I20240503 08:12:17.598258 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 299 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 298 index: 14371. Preceding OpId from leader: term: 299 index: 14372. (index mismatch)
I20240503 08:12:17.598513 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14372, Last known committed idx: 14371, Time since last communication: 0.000s
I20240503 08:12:17.599953 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 299 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 298 index: 14371. Preceding OpId from leader: term: 299 index: 14372. (index mismatch)
I20240503 08:12:17.600174 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14372, Last known committed idx: 14371, Time since last communication: 0.000s
W20240503 08:12:17.613201 26626 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 295 current term: 299 is higher.
I20240503 08:12:17.629998 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 299 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:17.630096 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 299 FOLLOWER]: Advancing to term 300
I20240503 08:12:17.630762 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 300 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.630900 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 300 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.631096 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 300 candidate_status { last_received { term: 299 index: 14372 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.631108 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 300 candidate_status { last_received { term: 299 index: 14372 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.631417 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 300 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.631549 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 300 FOLLOWER]: Leader election lost for term 300. Reason: could not achieve majority
I20240503 08:12:17.633517 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 300 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 299. Current term is 300. Ops: []
I20240503 08:12:17.633757 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 299.14372, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
I20240503 08:12:17.658910 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 299 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:17.658996 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 299 FOLLOWER]: Advancing to term 300
I20240503 08:12:17.659658 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 300 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.659828 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 300 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.660012 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 300 candidate_status { last_received { term: 299 index: 14372 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.660012 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 300 candidate_status { last_received { term: 299 index: 14372 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.660166 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 300 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 300: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:17.660328 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 300 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.660414 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 300 FOLLOWER]: Leader election lost for term 300. Reason: could not achieve majority
I20240503 08:12:17.663760 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 300 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 299. Current term is 300. Ops: []
I20240503 08:12:17.663985 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 299.14372, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
I20240503 08:12:17.666363 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 300 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.666416 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 300 FOLLOWER]: Advancing to term 301
I20240503 08:12:17.667143 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.667343 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 301 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.667438 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 301 candidate_status { last_received { term: 299 index: 14372 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.667472 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 301 candidate_status { last_received { term: 299 index: 14372 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.667529 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 300 FOLLOWER]: Advancing to term 301
I20240503 08:12:17.668308 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 301 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 301.
I20240503 08:12:17.668494 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 301 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.668601 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 FOLLOWER]: Leader election won for term 301
I20240503 08:12:17.668653 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.668710 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14372, Committed index: 14372, Last appended: 299.14372, Last appended by leader: 14372, Current term: 301, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.669201 26030 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 299 to 301, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 301 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:17.670349 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 299. Current term is 301. Ops: []
I20240503 08:12:17.670727 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 299.14372, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
I20240503 08:12:17.670830 26603 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 301.14373 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14372
I20240503 08:12:17.686195 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 301 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 299. Current term is 301. Ops: []
I20240503 08:12:17.686439 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 299.14372, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
W20240503 08:12:17.692364 26612 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 299 current term: 301 is higher.
I20240503 08:12:17.694295 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 299. Current term is 301. Ops: []
I20240503 08:12:17.694551 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 299.14372, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
I20240503 08:12:17.694660 26603 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 301.14373 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14372
I20240503 08:12:17.695456 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 301 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 299 index: 14372. Preceding OpId from leader: term: 301 index: 14373. (index mismatch)
I20240503 08:12:17.695655 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
W20240503 08:12:17.697525 26626 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 295 current term: 299 is higher.
I20240503 08:12:17.702945 26134 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 LEADER]: Stepping down as leader of term 299
I20240503 08:12:17.703002 26134 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:17.703048 26134 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14372, Committed index: 14372, Last appended: 299.14372, Last appended by leader: 14372, Current term: 299, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.703155 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 299 FOLLOWER]: Advancing to term 301
I20240503 08:12:17.703786 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 301 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 299 index: 14372. Preceding OpId from leader: term: 301 index: 14373. (index mismatch)
I20240503 08:12:17.704008 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14373, Last known committed idx: 14372, Time since last communication: 0.000s
I20240503 08:12:17.727005 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 301 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.727125 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 301 FOLLOWER]: Advancing to term 302
I20240503 08:12:17.727880 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.728070 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 302 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.728266 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 302 candidate_status { last_received { term: 301 index: 14373 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.728266 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 302 candidate_status { last_received { term: 301 index: 14373 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.728546 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 302 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.728660 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Leader election lost for term 302. Reason: could not achieve majority
I20240503 08:12:17.732667 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 301. Current term is 302. Ops: []
I20240503 08:12:17.732913 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 301.14373, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.734953 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 301 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.735005 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 301 FOLLOWER]: Advancing to term 302
I20240503 08:12:17.735656 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 302 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.735821 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 302 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.735877 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 302 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 301. Current term is 302. Ops: []
I20240503 08:12:17.735957 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 302 candidate_status { last_received { term: 301 index: 14373 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.736110 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 302 candidate_status { last_received { term: 301 index: 14373 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.736204 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 302: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:17.736341 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 302 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.736410 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 301.14373, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.736420 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 302 FOLLOWER]: Leader election lost for term 302. Reason: could not achieve majority
I20240503 08:12:17.759371 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 301. Current term is 302. Ops: []
I20240503 08:12:17.759675 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 301.14373, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.767239 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.767355 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 302 FOLLOWER]: Advancing to term 303
I20240503 08:12:17.768110 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.768280 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 303 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.768491 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 303 candidate_status { last_received { term: 301 index: 14373 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.768532 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 302 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 301. Current term is 302. Ops: []
I20240503 08:12:17.768641 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 303 candidate_status { last_received { term: 301 index: 14373 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.768710 26134 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 302 FOLLOWER]: Advancing to term 303
I20240503 08:12:17.768751 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 301.14373, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.769331 26134 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 303 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 303.
I20240503 08:12:17.769518 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 303 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:17.769608 26627 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 FOLLOWER]: Leader election won for term 303
I20240503 08:12:17.769667 26627 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:17.769735 26627 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14373, Committed index: 14373, Last appended: 301.14373, Last appended by leader: 14373, Current term: 303, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.770210 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 301 to 303, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 303 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:17.786352 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 LEADER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 301. Current term is 303. Ops: []
I20240503 08:12:17.786630 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 301.14373, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.786746 26611 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 303.14374 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14373
I20240503 08:12:17.786974 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 LEADER]: Stepping down as leader of term 301
I20240503 08:12:17.787004 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.787057 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14373, Committed index: 14373, Last appended: 301.14373, Last appended by leader: 14373, Current term: 301, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.787137 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 301 FOLLOWER]: Advancing to term 302
I20240503 08:12:17.793630 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 302 FOLLOWER]: Advancing to term 303
I20240503 08:12:17.794288 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 303 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 301 index: 14373. Preceding OpId from leader: term: 303 index: 14374. (index mismatch)
I20240503 08:12:17.794525 26627 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.806052 26134 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 303 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 301 index: 14373. Preceding OpId from leader: term: 303 index: 14374. (index mismatch)
I20240503 08:12:17.806365 26627 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14374, Last known committed idx: 14373, Time since last communication: 0.000s
I20240503 08:12:17.837829 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 303 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:17.837939 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 303 FOLLOWER]: Advancing to term 304
I20240503 08:12:17.838660 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 304 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.838850 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 304 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.839084 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 304 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.839124 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 304 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.839346 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 304 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.839453 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 304 FOLLOWER]: Leader election lost for term 304. Reason: could not achieve majority
I20240503 08:12:17.841274 26134 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 304 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 303. Current term is 304. Ops: []
I20240503 08:12:17.841516 26634 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 303.14374, Next index: 14375, Last known committed idx: 14374, Time since last communication: 0.000s
I20240503 08:12:17.870839 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 304 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.870937 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 304 FOLLOWER]: Advancing to term 305
I20240503 08:12:17.871618 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.871762 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 305 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.871974 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 305 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.871974 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 305 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.872287 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 305 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:17.872380 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Leader election lost for term 305. Reason: could not achieve majority
I20240503 08:12:17.875429 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 303 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:17.875500 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 303 FOLLOWER]: Advancing to term 304
I20240503 08:12:17.876094 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 304 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.876248 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 304 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.876385 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 304 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.876431 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 304 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.876482 26134 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 304. Current term is 305.
I20240503 08:12:17.876544 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 303. Current term is 305. Ops: []
I20240503 08:12:17.876756 26176 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 304 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 304. Current term is 305.
I20240503 08:12:17.876792 26176 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 304 election: Cancelling election due to peer responding with higher term
I20240503 08:12:17.876811 26634 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 303.14374, Next index: 14375, Last known committed idx: 14374, Time since last communication: 0.000s
I20240503 08:12:17.876884 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 304 FOLLOWER]: Advancing to term 305
I20240503 08:12:17.877445 26611 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 305 FOLLOWER]: Leader election lost for term 304. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 304. Current term is 305.
I20240503 08:12:17.881752 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 305 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 303. Current term is 305. Ops: []
I20240503 08:12:17.881964 26517 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 303.14374, Next index: 14375, Last known committed idx: 14374, Time since last communication: 0.000s
I20240503 08:12:17.893837 26627 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 LEADER]: Stepping down as leader of term 303
I20240503 08:12:17.893921 26627 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:17.893998 26627 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14374, Committed index: 14374, Last appended: 303.14374, Last appended by leader: 14374, Current term: 303, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.894131 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 303 FOLLOWER]: Advancing to term 304
I20240503 08:12:17.918040 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 305 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.918140 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 305 FOLLOWER]: Advancing to term 306
I20240503 08:12:17.918722 26634 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 304 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:17.918784 26634 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 304 FOLLOWER]: Advancing to term 305
I20240503 08:12:17.918906 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.919072 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 306 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:17.919289 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 306 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.919289 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 306 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:17.919407 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 305 FOLLOWER]: Advancing to term 306
I20240503 08:12:17.919598 26634 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 305 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.919713 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 305 FOLLOWER]: Advancing to term 306
I20240503 08:12:17.919802 26634 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 305 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.919862 26134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 305 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.920013 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 306 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 306.
I20240503 08:12:17.920185 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 306 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:17.920213 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 305 candidate_status { last_received { term: 303 index: 14374 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.920279 26210 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 305. Current term is 306.
I20240503 08:12:17.920312 26611 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 FOLLOWER]: Leader election won for term 306
I20240503 08:12:17.920369 26611 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:17.920428 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 306 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 306.
I20240503 08:12:17.920430 26611 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14374, Committed index: 14374, Last appended: 303.14374, Last appended by leader: 14374, Current term: 306, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.920619 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 305 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 305. Current term is 306.
I20240503 08:12:17.920660 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 305 election: Cancelling election due to peer responding with higher term
I20240503 08:12:17.920733 26634 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 306 FOLLOWER]: Leader election lost for term 305. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 305. Current term is 306.
I20240503 08:12:17.920899 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 303 to 306, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 306 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
W20240503 08:12:17.922158 26378 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 302 current term: 306 is higher.
I20240503 08:12:17.945716 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 306 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 303 index: 14374. Preceding OpId from leader: term: 306 index: 14375. (index mismatch)
I20240503 08:12:17.946105 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14375, Last known committed idx: 14374, Time since last communication: 0.000s
I20240503 08:12:17.950767 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 306 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 303 index: 14374. Preceding OpId from leader: term: 306 index: 14375. (index mismatch)
I20240503 08:12:17.951099 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14375, Last known committed idx: 14374, Time since last communication: 0.000s
I20240503 08:12:17.977787 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 306 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:17.977895 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 306 FOLLOWER]: Advancing to term 307
I20240503 08:12:17.978682 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 307 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:17.978801 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 307 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 306. Current term is 307. Ops: []
I20240503 08:12:17.979003 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 307 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:17.979141 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 307 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:17.979003 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 307 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:17.979105 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 306.14375, Next index: 14376, Last known committed idx: 14375, Time since last communication: 0.000s
I20240503 08:12:17.979338 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 307 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:17.979449 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 307 FOLLOWER]: Leader election lost for term 307. Reason: could not achieve majority
W20240503 08:12:17.999557 26611 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 302 current term: 306 is higher.
I20240503 08:12:18.011407 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 307 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 306. Current term is 307. Ops: []
I20240503 08:12:18.011790 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 306.14375, Next index: 14376, Last known committed idx: 14375, Time since last communication: 0.000s
I20240503 08:12:18.018249 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 307 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.018353 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 307 FOLLOWER]: Advancing to term 308
I20240503 08:12:18.019102 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.019273 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 308 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.019501 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 308 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.019501 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 308 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.019758 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 308 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.019910 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Leader election lost for term 308. Reason: could not achieve majority
W20240503 08:12:18.024789 26611 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 303 current term: 306 is higher.
I20240503 08:12:18.038836 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 306 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.038944 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 306 FOLLOWER]: Advancing to term 307
I20240503 08:12:18.039764 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 307 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.039918 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 307 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 306. Current term is 307. Ops: []
I20240503 08:12:18.039973 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 307 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.040167 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 307 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.040167 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 307 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.040185 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 306.14375, Next index: 14376, Last known committed idx: 14375, Time since last communication: 0.000s
I20240503 08:12:18.040298 26284 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 307. Current term is 308.
I20240503 08:12:18.040419 26097 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 307 election: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 307. Current term is 308.
I20240503 08:12:18.040458 26097 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 307 election: Cancelling election due to peer responding with higher term
I20240503 08:12:18.040541 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 307 FOLLOWER]: Advancing to term 308
I20240503 08:12:18.041127 26626 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 308 FOLLOWER]: Leader election lost for term 307. Reason: Vote denied by peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 for earlier term 307. Current term is 308.
I20240503 08:12:18.044234 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 306. Current term is 308. Ops: []
I20240503 08:12:18.044481 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 306.14375, Next index: 14376, Last known committed idx: 14375, Time since last communication: 0.000s
I20240503 08:12:18.051420 26611 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 LEADER]: Stepping down as leader of term 306
I20240503 08:12:18.051510 26611 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.051592 26611 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14375, Committed index: 14375, Last appended: 306.14375, Last appended by leader: 14375, Current term: 306, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.051720 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 306 FOLLOWER]: Advancing to term 307
I20240503 08:12:18.073675 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 308 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.073786 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 308 FOLLOWER]: Advancing to term 309
I20240503 08:12:18.074560 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 309 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.074766 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 309 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.075008 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 309 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.075008 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 309 candidate_status { last_received { term: 306 index: 14375 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.075142 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 308 FOLLOWER]: Advancing to term 309
I20240503 08:12:18.075131 26210 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 307 FOLLOWER]: Advancing to term 309
I20240503 08:12:18.075847 26210 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 309 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 309.
I20240503 08:12:18.075847 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 309 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 309.
I20240503 08:12:18.076104 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 309 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:18.076239 26626 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 309 FOLLOWER]: Leader election won for term 309
I20240503 08:12:18.076300 26626 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 309 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:18.076445 26626 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14375, Committed index: 14375, Last appended: 306.14375, Last appended by leader: 14375, Current term: 309, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.076907 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 306 to 309, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 309 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:18.103411 26210 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 309 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 306 index: 14375. Preceding OpId from leader: term: 309 index: 14376. (index mismatch)
I20240503 08:12:18.103713 26626 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14376, Last known committed idx: 14375, Time since last communication: 0.000s
I20240503 08:12:18.111447 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 309 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 306 index: 14375. Preceding OpId from leader: term: 309 index: 14376. (index mismatch)
I20240503 08:12:18.111709 26626 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14376, Last known committed idx: 14375, Time since last communication: 0.000s
I20240503 08:12:18.135325 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 309 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:18.135425 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 309 FOLLOWER]: Advancing to term 310
I20240503 08:12:18.136200 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 310 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.136389 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 310 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.136656 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 310 candidate_status { last_received { term: 309 index: 14376 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.136660 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 310 candidate_status { last_received { term: 309 index: 14376 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.136994 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 310 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.137126 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 310 FOLLOWER]: Leader election lost for term 310. Reason: could not achieve majority
I20240503 08:12:18.137859 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 310 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 309. Current term is 310. Ops: []
I20240503 08:12:18.138129 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 309.14376, Next index: 14377, Last known committed idx: 14376, Time since last communication: 0.000s
I20240503 08:12:18.142136 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 309 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:18.142223 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 309 FOLLOWER]: Advancing to term 310
I20240503 08:12:18.142985 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 310 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.143152 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 310 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.143361 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 310 candidate_status { last_received { term: 309 index: 14376 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.143361 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 310 candidate_status { last_received { term: 309 index: 14376 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.143474 26210 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 310 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 310: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:18.143618 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 310 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.143734 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 310 FOLLOWER]: Leader election lost for term 310. Reason: could not achieve majority
I20240503 08:12:18.147616 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 310 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 309. Current term is 310. Ops: []
I20240503 08:12:18.147890 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 309.14376, Next index: 14377, Last known committed idx: 14376, Time since last communication: 0.000s
I20240503 08:12:18.170828 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 310 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.170938 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 310 FOLLOWER]: Advancing to term 311
I20240503 08:12:18.171660 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.171830 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 311 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.172040 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 311 candidate_status { last_received { term: 309 index: 14376 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.172040 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 311 candidate_status { last_received { term: 309 index: 14376 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.172156 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 310 FOLLOWER]: Advancing to term 311
I20240503 08:12:18.172798 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 311 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 311.
I20240503 08:12:18.173000 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 311 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.173122 26612 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 FOLLOWER]: Leader election won for term 311
I20240503 08:12:18.173177 26612 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.173233 26612 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14376, Committed index: 14376, Last appended: 309.14376, Last appended by leader: 14376, Current term: 311, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
W20240503 08:12:18.173374 26348 lock_manager.cc:494] Txn 9 has not acquired the partition lock after 1000ms
I20240503 08:12:18.173439 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 309. Current term is 311. Ops: []
I20240503 08:12:18.173759 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 309.14376, Next index: 14377, Last known committed idx: 14376, Time since last communication: 0.000s
I20240503 08:12:18.173733 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 309 to 311, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 311 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:18.173842 26626 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 311.14377 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14376
W20240503 08:12:18.177995 26611 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 308 current term: 311 is higher.
I20240503 08:12:18.178220 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 311 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 309. Current term is 311. Ops: []
I20240503 08:12:18.178411 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 309.14376, Next index: 14377, Last known committed idx: 14376, Time since last communication: 0.000s
I20240503 08:12:18.197067 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 311 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 309 index: 14376. Preceding OpId from leader: term: 311 index: 14377. (index mismatch)
I20240503 08:12:18.197335 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14377, Last known committed idx: 14376, Time since last communication: 0.000s
I20240503 08:12:18.199744 26210 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 309. Current term is 311. Ops: []
I20240503 08:12:18.199987 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 309.14376, Next index: 14377, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.200066 26603 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer 7fb2b6702b7f495b92a4d8643d97b58d log is divergent from this leader: its last log entry 311.14377 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14377
I20240503 08:12:18.201234 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 309 LEADER]: Stepping down as leader of term 309
I20240503 08:12:18.201275 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 309 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:18.201318 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14376, Committed index: 14376, Last appended: 309.14376, Last appended by leader: 14376, Current term: 309, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.201408 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 309 FOLLOWER]: Advancing to term 311
I20240503 08:12:18.202034 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 311 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 309 index: 14376. Preceding OpId from leader: term: 311 index: 14377. (index mismatch)
I20240503 08:12:18.202286 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14377, Last known committed idx: 14376, Time since last communication: 0.000s
I20240503 08:12:18.229219 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 311 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.229321 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 311 FOLLOWER]: Advancing to term 312
I20240503 08:12:18.229988 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.230228 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 312 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.230415 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 312 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.230430 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 311. Current term is 312. Ops: []
I20240503 08:12:18.230415 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 312 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.230752 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 312 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.230765 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.230906 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Leader election lost for term 312. Reason: could not achieve majority
I20240503 08:12:18.233062 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 311 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.233131 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 311 FOLLOWER]: Advancing to term 312
I20240503 08:12:18.233786 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 312 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.234177 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 312 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.234302 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 312 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 311. Current term is 312. Ops: []
I20240503 08:12:18.234336 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 312 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.234356 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 312 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.234537 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.234763 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 312: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:18.234946 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 312 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.235086 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 312 FOLLOWER]: Leader election lost for term 312. Reason: could not achieve majority
I20240503 08:12:18.253958 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 311. Current term is 312. Ops: []
I20240503 08:12:18.254262 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.265384 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 312 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 311. Current term is 312. Ops: []
I20240503 08:12:18.265676 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.269452 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.269522 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 312 FOLLOWER]: Advancing to term 313
I20240503 08:12:18.269620 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 312 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.269680 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 312 FOLLOWER]: Advancing to term 313
I20240503 08:12:18.270259 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 313 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.270268 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 313 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.270507 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 313 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.270658 26210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 313 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.270653 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 313 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.270695 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 313 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.270776 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 313 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 313: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:18.270875 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 313 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.270916 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 313 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.270653 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 313 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.270994 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 313 FOLLOWER]: Leader election lost for term 313. Reason: could not achieve majority
I20240503 08:12:18.271059 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 313 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 313: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:18.271206 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 313 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.271289 26603 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 313 FOLLOWER]: Leader election lost for term 313. Reason: could not achieve majority
I20240503 08:12:18.277730 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 LEADER]: Stepping down as leader of term 311
I20240503 08:12:18.277793 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.277850 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14377, Committed index: 14377, Last appended: 311.14377, Last appended by leader: 14377, Current term: 311, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.277974 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 311 FOLLOWER]: Advancing to term 312
I20240503 08:12:18.301808 26603 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 313 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.301915 26603 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 313 FOLLOWER]: Advancing to term 314
I20240503 08:12:18.302649 26603 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 314 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.302803 26603 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 314 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.303018 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 314 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.303027 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 314 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.303102 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 312 FOLLOWER]: Advancing to term 314
I20240503 08:12:18.303146 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 313 FOLLOWER]: Advancing to term 314
I20240503 08:12:18.303675 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 314 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 314.
I20240503 08:12:18.303717 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 314 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 314.
I20240503 08:12:18.303872 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 314 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:18.303972 26603 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 314 FOLLOWER]: Leader election won for term 314
I20240503 08:12:18.304031 26603 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 314 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:18.304104 26603 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14377, Committed index: 14377, Last appended: 311.14377, Last appended by leader: 14377, Current term: 314, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.304517 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 311 to 314, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 314 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:18.326946 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 314 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 311 index: 14377. Preceding OpId from leader: term: 314 index: 14378. (index mismatch)
I20240503 08:12:18.327220 26603 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
W20240503 08:12:18.335737 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 311 current term: 314 is higher.
I20240503 08:12:18.338795 26611 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 314 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.338886 26611 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 314 FOLLOWER]: Advancing to term 315
I20240503 08:12:18.339550 26611 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.339725 26611 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 315 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.339890 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 315 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.339958 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 315. Ops: []
I20240503 08:12:18.339964 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 315 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.340214 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.340325 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 315 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.340445 26611 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Leader election lost for term 315. Reason: could not achieve majority
W20240503 08:12:18.356971 26515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 311 current term: 314 is higher.
I20240503 08:12:18.358768 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 314 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:18.358841 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 314 FOLLOWER]: Advancing to term 315
I20240503 08:12:18.359537 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 315 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.359730 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 315 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.359933 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 315 candidate_status { last_received { term: 314 index: 14378 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.359933 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 315 candidate_status { last_received { term: 314 index: 14378 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.360044 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 315: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:18.360214 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 315 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.360330 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 315 FOLLOWER]: Leader election lost for term 315. Reason: could not achieve majority
I20240503 08:12:18.361212 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 315 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 315. Ops: []
I20240503 08:12:18.361441 26626 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 314.14378, Next index: 14379, Last known committed idx: 14378, Time since last communication: 0.000s
I20240503 08:12:18.363880 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 315. Ops: [314.14378-314.14378]
I20240503 08:12:18.364127 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.378705 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 315. Ops: [314.14378-314.14378]
I20240503 08:12:18.379009 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.381289 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.381366 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 315 FOLLOWER]: Advancing to term 316
I20240503 08:12:18.382030 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 316 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.382223 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 316 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.382409 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 316 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.382412 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 316 candidate_status { last_received { term: 311 index: 14377 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.382537 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 315 FOLLOWER]: Advancing to term 316
I20240503 08:12:18.383106 26284 raft_consensus.cc:2396] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 316 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d for term 316 because replica has last-logged OpId of term: 314 index: 14378, which is greater than that of the candidate, which has last-logged OpId of term: 311 index: 14377.
I20240503 08:12:18.383280 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 316 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.383368 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 316 FOLLOWER]: Leader election lost for term 316. Reason: could not achieve majority
I20240503 08:12:18.385581 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 316 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 316. Ops: []
I20240503 08:12:18.385883 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 314.14378, Next index: 14379, Last known committed idx: 14378, Time since last communication: 0.000s
I20240503 08:12:18.391983 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 316 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.392051 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 316 FOLLOWER]: Advancing to term 317
I20240503 08:12:18.392654 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.392805 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 317 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.392956 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 317 candidate_status { last_received { term: 314 index: 14378 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.392980 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 317 candidate_status { last_received { term: 314 index: 14378 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.393065 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 316 FOLLOWER]: Advancing to term 317
I20240503 08:12:18.393580 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 317 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 317.
I20240503 08:12:18.393774 26249 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 317 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa; no voters: db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.393884 26627 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 FOLLOWER]: Leader election won for term 317
I20240503 08:12:18.393959 26627 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:18.394022 26627 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14378, Committed index: 14378, Last appended: 314.14378, Last appended by leader: 14378, Current term: 317, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.394481 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 314 to 317, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 317 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:18.409725 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 317 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 317. Ops: [314.14378-314.14378]
I20240503 08:12:18.409912 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 LEADER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 314. Current term is 317. Ops: []
I20240503 08:12:18.410018 26603 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 311.14377, Next index: 14378, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.410130 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 314.14378, Next index: 14379, Last known committed idx: 14378, Time since last communication: 0.000s
I20240503 08:12:18.410233 26515 consensus_queue.cc:1225] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer e8eb2ae79c07480d86c3ecf1261153aa log is divergent from this leader: its last log entry 317.14379 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14378
I20240503 08:12:18.427677 26133 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 314 LEADER]: Stepping down as leader of term 314
I20240503 08:12:18.427772 26133 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 314 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:18.427827 26133 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14378, Committed index: 14378, Last appended: 314.14378, Last appended by leader: 14378, Current term: 314, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.427851 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 317 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 311 index: 14377. Preceding OpId from leader: term: 317 index: 14379. (index mismatch)
I20240503 08:12:18.427954 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 314 FOLLOWER]: Advancing to term 317
I20240503 08:12:18.428143 26627 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14379, Last known committed idx: 14377, Time since last communication: 0.000s
I20240503 08:12:18.428808 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 317 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 314 index: 14378. Preceding OpId from leader: term: 317 index: 14379. (index mismatch)
I20240503 08:12:18.429064 26627 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14379, Last known committed idx: 14378, Time since last communication: 0.000s
I20240503 08:12:18.483804 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 317 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:18.483894 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 317 FOLLOWER]: Advancing to term 318
I20240503 08:12:18.484622 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.484794 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 318 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.484975 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 318 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.485016 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 318 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.485291 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 318 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.485399 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Leader election lost for term 318. Reason: could not achieve majority
I20240503 08:12:18.490264 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 317. Current term is 318. Ops: []
I20240503 08:12:18.490526 26634 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 317.14379, Next index: 14380, Last known committed idx: 14379, Time since last communication: 0.000s
I20240503 08:12:18.506645 26515 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 317 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:18.506765 26515 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 317 FOLLOWER]: Advancing to term 318
I20240503 08:12:18.507655 26515 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 318 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.507879 26515 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 318 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.508137 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 318 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.508136 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 318 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.508250 26209 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 318: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:18.508437 26098 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 318 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.508551 26515 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 318 FOLLOWER]: Leader election lost for term 318. Reason: could not achieve majority
I20240503 08:12:18.510915 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 318 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 317. Current term is 318. Ops: []
I20240503 08:12:18.511143 26634 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 317.14379, Next index: 14380, Last known committed idx: 14379, Time since last communication: 0.000s
I20240503 08:12:18.520201 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 317. Current term is 318. Ops: []
I20240503 08:12:18.520510 26627 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 317.14379, Next index: 14380, Last known committed idx: 14379, Time since last communication: 0.000s
I20240503 08:12:18.530037 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.530135 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 318 FOLLOWER]: Advancing to term 319
I20240503 08:12:18.530894 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 319 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.531069 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 319 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.531265 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 319 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.531270 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 319 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.531360 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 318 FOLLOWER]: Advancing to term 319
I20240503 08:12:18.531948 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 319 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 319.
I20240503 08:12:18.532143 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 319 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters: e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.532241 26612 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 319 FOLLOWER]: Leader election won for term 319
I20240503 08:12:18.532305 26612 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 319 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.532369 26612 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14379, Committed index: 14379, Last appended: 317.14379, Last appended by leader: 14379, Current term: 319, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.532809 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 317 to 319, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 319 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:18.533746 26638 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 LEADER]: Stepping down as leader of term 317
I20240503 08:12:18.533788 26638 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:18.533864 26638 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14379, Committed index: 14379, Last appended: 317.14379, Last appended by leader: 14379, Current term: 317, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.533951 26638 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 317 FOLLOWER]: Advancing to term 318
I20240503 08:12:18.563284 26634 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 318 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.563392 26634 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 318 FOLLOWER]: Advancing to term 319
I20240503 08:12:18.564175 26634 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 319 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.564338 26634 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 319 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.564643 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 319 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.564653 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 319 candidate_status { last_received { term: 317 index: 14379 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.564772 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 319 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa in current term 319: Already voted for candidate 7fb2b6702b7f495b92a4d8643d97b58d in this term.
I20240503 08:12:18.564937 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 319 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.565073 26634 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 319 FOLLOWER]: Leader election lost for term 319. Reason: could not achieve majority
I20240503 08:12:18.568287 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 319 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 317 index: 14379. Preceding OpId from leader: term: 319 index: 14380. (index mismatch)
I20240503 08:12:18.568423 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 319 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 317 index: 14379. Preceding OpId from leader: term: 319 index: 14380. (index mismatch)
I20240503 08:12:18.568471 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14380, Last known committed idx: 14379, Time since last communication: 0.000s
I20240503 08:12:18.568614 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14380, Last known committed idx: 14379, Time since last communication: 0.000s
I20240503 08:12:18.599857 26634 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 319 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.599946 26634 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 319 FOLLOWER]: Advancing to term 320
I20240503 08:12:18.600633 26634 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.600800 26634 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 320 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.600979 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 320 candidate_status { last_received { term: 319 index: 14380 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.601037 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 320 candidate_status { last_received { term: 319 index: 14380 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.601269 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 320 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:18.601363 26634 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Leader election lost for term 320. Reason: could not achieve majority
I20240503 08:12:18.605477 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 319. Current term is 320. Ops: []
I20240503 08:12:18.605753 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 319.14380, Next index: 14381, Last known committed idx: 14380, Time since last communication: 0.000s
I20240503 08:12:18.629495 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 319 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.629614 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 319 FOLLOWER]: Advancing to term 320
I20240503 08:12:18.630322 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 320 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.630491 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 320 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.630733 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 320 candidate_status { last_received { term: 319 index: 14380 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.630810 26284 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Leader election vote request: Denying vote to candidate db1737abb1d5409bb049fa1c94d3ed48 in current term 320: Already voted for candidate e8eb2ae79c07480d86c3ecf1261153aa in this term.
I20240503 08:12:18.630739 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 320 candidate_status { last_received { term: 319 index: 14380 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.631068 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 320 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.631170 26626 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 320 FOLLOWER]: Leader election lost for term 320. Reason: could not achieve majority
I20240503 08:12:18.632476 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 320 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 319. Current term is 320. Ops: []
I20240503 08:12:18.632680 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 319.14380, Next index: 14381, Last known committed idx: 14380, Time since last communication: 0.000s
I20240503 08:12:18.636096 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 319. Current term is 320. Ops: []
I20240503 08:12:18.636235 26638 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.636281 26638 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 320 FOLLOWER]: Advancing to term 321
I20240503 08:12:18.636351 26378 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 319.14380, Next index: 14381, Last known committed idx: 14380, Time since last communication: 0.000s
I20240503 08:12:18.636993 26638 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 321 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.637120 26638 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 321 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.637295 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 321 candidate_status { last_received { term: 319 index: 14380 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.637303 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 321 candidate_status { last_received { term: 319 index: 14380 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.637395 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 320 FOLLOWER]: Advancing to term 321
I20240503 08:12:18.637917 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 321 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 321.
I20240503 08:12:18.638113 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 321 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d
I20240503 08:12:18.638226 26638 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 321 FOLLOWER]: Leader election won for term 321
I20240503 08:12:18.638285 26638 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 321 LEADER]: Becoming Leader. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:18.638342 26638 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14380, Committed index: 14380, Last appended: 319.14380, Last appended by leader: 14380, Current term: 321, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.638828 26029 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa reported cstate change: term changed from 319 to 321, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195). New cstate: current_term: 321 leader_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:18.651921 26378 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 319 LEADER]: Stepping down as leader of term 319
I20240503 08:12:18.652022 26378 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 319 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.652122 26378 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14380, Committed index: 14380, Last appended: 319.14380, Last appended by leader: 14380, Current term: 319, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.652274 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 319 FOLLOWER]: Advancing to term 320
I20240503 08:12:18.662180 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 321 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 319 index: 14380. Preceding OpId from leader: term: 321 index: 14381. (index mismatch)
I20240503 08:12:18.662479 26638 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14381, Last known committed idx: 14380, Time since last communication: 0.000s
I20240503 08:12:18.664270 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 320 FOLLOWER]: Advancing to term 321
I20240503 08:12:18.664947 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 321 FOLLOWER]: Refusing update from remote peer e8eb2ae79c07480d86c3ecf1261153aa: Log matching property violated. Preceding OpId in replica: term: 319 index: 14380. Preceding OpId from leader: term: 321 index: 14381. (index mismatch)
I20240503 08:12:18.665220 26638 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14381, Last known committed idx: 14380, Time since last communication: 0.000s
I20240503 08:12:18.745733 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 321 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:18.745833 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 321 FOLLOWER]: Advancing to term 322
I20240503 08:12:18.746512 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 322 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.746694 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 322 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.746898 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 322 candidate_status { last_received { term: 321 index: 14381 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.746903 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 322 candidate_status { last_received { term: 321 index: 14381 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.747217 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 322 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.747315 26626 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 322 FOLLOWER]: Leader election lost for term 322. Reason: could not achieve majority
I20240503 08:12:18.747392 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 322 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 321. Current term is 322. Ops: []
I20240503 08:12:18.747601 26627 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 321.14381, Next index: 14382, Last known committed idx: 14381, Time since last communication: 0.000s
I20240503 08:12:18.752794 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 321 FOLLOWER]: Starting leader election (detected failure of leader e8eb2ae79c07480d86c3ecf1261153aa)
I20240503 08:12:18.752857 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 321 FOLLOWER]: Advancing to term 322
I20240503 08:12:18.753492 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 322 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.753645 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 322 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.753839 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 322 candidate_status { last_received { term: 321 index: 14381 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.753839 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 322 candidate_status { last_received { term: 321 index: 14381 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.753930 26133 raft_consensus.cc:2379] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 322 FOLLOWER]: Leader election vote request: Denying vote to candidate 7fb2b6702b7f495b92a4d8643d97b58d in current term 322: Already voted for candidate db1737abb1d5409bb049fa1c94d3ed48 in this term.
I20240503 08:12:18.754153 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 322 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.754276 26378 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 322 FOLLOWER]: Leader election lost for term 322. Reason: could not achieve majority
I20240503 08:12:18.756704 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 322 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 321. Current term is 322. Ops: []
I20240503 08:12:18.756911 26638 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 321.14381, Next index: 14382, Last known committed idx: 14381, Time since last communication: 0.000s
I20240503 08:12:18.783859 26627 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 321 LEADER]: Stepping down as leader of term 321
I20240503 08:12:18.783965 26627 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 321 LEADER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Running, Role: LEADER
I20240503 08:12:18.783990 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 322 FOLLOWER]: Rejecting Update request from peer e8eb2ae79c07480d86c3ecf1261153aa for earlier term 321. Current term is 322. Ops: []
I20240503 08:12:18.784046 26627 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14381, Committed index: 14381, Last appended: 321.14381, Last appended by leader: 14381, Current term: 321, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.784168 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 321 FOLLOWER]: Advancing to term 322
I20240503 08:12:18.789085 26378 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 322 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.789171 26378 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 322 FOLLOWER]: Advancing to term 323
I20240503 08:12:18.789784 26378 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 323 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.789935 26378 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 323 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.790144 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 323 candidate_status { last_received { term: 321 index: 14381 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.790155 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 323 candidate_status { last_received { term: 321 index: 14381 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.790226 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 322 FOLLOWER]: Advancing to term 323
I20240503 08:12:18.790226 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 322 FOLLOWER]: Advancing to term 323
I20240503 08:12:18.790946 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 323 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 323.
I20240503 08:12:18.790946 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 323 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 323.
I20240503 08:12:18.791205 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 323 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:18.791283 26378 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 323 FOLLOWER]: Leader election won for term 323
I20240503 08:12:18.791370 26378 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 323 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.791435 26378 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14381, Committed index: 14381, Last appended: 321.14381, Last appended by leader: 14381, Current term: 323, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.791900 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 321 to 323, leader changed from e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 323 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:18.814333 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 323 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 321 index: 14381. Preceding OpId from leader: term: 323 index: 14382. (index mismatch)
I20240503 08:12:18.814673 26378 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14382, Last known committed idx: 14381, Time since last communication: 0.000s
I20240503 08:12:18.817982 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 323 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 321 index: 14381. Preceding OpId from leader: term: 323 index: 14382. (index mismatch)
I20240503 08:12:18.818313 26611 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14382, Last known committed idx: 14381, Time since last communication: 0.000s
I20240503 08:12:18.849900 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 323 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.850010 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 323 FOLLOWER]: Advancing to term 324
I20240503 08:12:18.850751 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 324 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.850921 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 324 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.851176 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 324 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.851191 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 324 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.851498 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 324 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.851617 26626 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 324 FOLLOWER]: Leader election lost for term 324. Reason: could not achieve majority
I20240503 08:12:18.854995 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 324 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 323. Current term is 324. Ops: []
I20240503 08:12:18.855266 26611 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 323.14382, Next index: 14383, Last known committed idx: 14382, Time since last communication: 0.000s
I20240503 08:12:18.886721 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 324 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.886830 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 324 FOLLOWER]: Advancing to term 325
I20240503 08:12:18.887665 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.887782 26133 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 323. Current term is 325. Ops: []
I20240503 08:12:18.887933 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 325 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.887944 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 325 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.887950 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 325 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.888039 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: INVALID_TERM, Last received: 323.14382, Next index: 14383, Last known committed idx: 14382, Time since last communication: 0.000s
I20240503 08:12:18.888208 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 325 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.888291 26626 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Leader election lost for term 325. Reason: could not achieve majority
I20240503 08:12:18.894026 26612 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 323 LEADER]: Stepping down as leader of term 323
I20240503 08:12:18.894086 26612 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 323 LEADER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:18.894143 26612 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14382, Committed index: 14382, Last appended: 323.14382, Last appended by leader: 14382, Current term: 323, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.894237 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 323 FOLLOWER]: Advancing to term 324
I20240503 08:12:18.903591 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 323 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:18.903687 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 323 FOLLOWER]: Advancing to term 324
I20240503 08:12:18.905143 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 324 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.905304 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 324 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.905530 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 324 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.905535 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 324 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.905647 26133 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 324. Current term is 325.
I20240503 08:12:18.905833 26251 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 324 election: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 324. Current term is 325.
I20240503 08:12:18.905881 26251 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 324 election: Cancelling election due to peer responding with higher term
I20240503 08:12:18.906010 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 324 FOLLOWER]: Advancing to term 325
I20240503 08:12:18.906231 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 324 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 324.
I20240503 08:12:18.906672 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 325 FOLLOWER]: Leader election lost for term 324. Reason: Vote denied by peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 324. Current term is 325.
I20240503 08:12:18.923002 26626 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:18.923110 26626 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 325 FOLLOWER]: Advancing to term 326
I20240503 08:12:18.923787 26626 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 326 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.923980 26626 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 326 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:18.924199 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 326 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.924261 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 325 FOLLOWER]: Advancing to term 326
I20240503 08:12:18.924227 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "db1737abb1d5409bb049fa1c94d3ed48" candidate_term: 326 candidate_status { last_received { term: 323 index: 14382 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:18.924324 26209 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 324 FOLLOWER]: Advancing to term 326
I20240503 08:12:18.924835 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 326 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 326.
I20240503 08:12:18.924938 26209 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 326 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1737abb1d5409bb049fa1c94d3ed48 in term 326.
I20240503 08:12:18.925060 26097 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 326 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa; no voters:
I20240503 08:12:18.925181 26626 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 326 FOLLOWER]: Leader election won for term 326
I20240503 08:12:18.925244 26626 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 326 LEADER]: Becoming Leader. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:18.925305 26626 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14382, Committed index: 14382, Last appended: 323.14382, Last appended by leader: 14382, Current term: 326, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.925798 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 reported cstate change: term changed from 323 to 326, leader changed from 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194) to db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193). New cstate: current_term: 326 leader_uuid: "db1737abb1d5409bb049fa1c94d3ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:18.951592 26209 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 326 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 323 index: 14382. Preceding OpId from leader: term: 326 index: 14383. (index mismatch)
I20240503 08:12:18.952086 26626 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14383, Last known committed idx: 14382, Time since last communication: 0.000s
I20240503 08:12:18.961697 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 326 FOLLOWER]: Refusing update from remote peer db1737abb1d5409bb049fa1c94d3ed48: Log matching property violated. Preceding OpId in replica: term: 323 index: 14382. Preceding OpId from leader: term: 326 index: 14383. (index mismatch)
I20240503 08:12:18.962015 26626 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14383, Last known committed idx: 14382, Time since last communication: 0.000s
I20240503 08:12:18.983983 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 326 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:18.984093 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 326 FOLLOWER]: Advancing to term 327
I20240503 08:12:18.984903 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 327 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:18.985046 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 327 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:18.985196 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 327 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:18.985196 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 327 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:18.985411 26173 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 327 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:18.985482 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 327 FOLLOWER]: Leader election lost for term 327. Reason: could not achieve majority
I20240503 08:12:18.988409 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 327 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 326. Current term is 327. Ops: []
I20240503 08:12:18.988643 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 326.14383, Next index: 14384, Last known committed idx: 14383, Time since last communication: 0.000s
I20240503 08:12:19.019636 26209 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 327 FOLLOWER]: Rejecting Update request from peer db1737abb1d5409bb049fa1c94d3ed48 for earlier term 326. Current term is 327. Ops: []
I20240503 08:12:19.019943 26515 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 }, Status: INVALID_TERM, Last received: 326.14383, Next index: 14384, Last known committed idx: 14383, Time since last communication: 0.000s
I20240503 08:12:19.028024 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 327 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:19.028120 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 327 FOLLOWER]: Advancing to term 328
I20240503 08:12:19.028784 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.028928 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 328 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:19.029132 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 328 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:19.029182 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 328 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:19.029362 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 328 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:19.029451 26612 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Leader election lost for term 328. Reason: could not achieve majority
I20240503 08:12:19.040076 26639 raft_consensus.cc:3032] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 326 LEADER]: Stepping down as leader of term 326
I20240503 08:12:19.040167 26639 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 326 LEADER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Running, Role: LEADER
I20240503 08:12:19.040236 26639 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14383, Committed index: 14383, Last appended: 326.14383, Last appended by leader: 14383, Current term: 326, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.040377 26639 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 326 FOLLOWER]: Advancing to term 327
I20240503 08:12:19.066972 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 326 FOLLOWER]: Starting leader election (detected failure of leader db1737abb1d5409bb049fa1c94d3ed48)
I20240503 08:12:19.067082 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 326 FOLLOWER]: Advancing to term 327
I20240503 08:12:19.067818 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 327 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.067962 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 327 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:19.068182 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 327 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:19.068179 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 327 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:19.068360 26209 raft_consensus.cc:2354] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 327. Current term is 328.
I20240503 08:12:19.068529 26249 leader_election.cc:398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 327 election: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 327. Current term is 328.
I20240503 08:12:19.068589 26249 leader_election.cc:401] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 327 election: Cancelling election due to peer responding with higher term
I20240503 08:12:19.068670 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 327 FOLLOWER]: Advancing to term 328
I20240503 08:12:19.068868 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 327 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8eb2ae79c07480d86c3ecf1261153aa in term 327.
I20240503 08:12:19.069310 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 328 FOLLOWER]: Leader election lost for term 327. Reason: Vote denied by peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443) with higher term. Message: Invalid argument: T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Leader election vote request: Denying vote to candidate e8eb2ae79c07480d86c3ecf1261153aa for earlier term 327. Current term is 328.
I20240503 08:12:19.090456 26612 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:19.090562 26612 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 328 FOLLOWER]: Advancing to term 329
I20240503 08:12:19.091252 26612 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.091506 26612 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 329 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:19.091653 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 329 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:19.091636 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" candidate_term: 329 candidate_status { last_received { term: 326 index: 14383 } } ignore_live_leader: false dest_uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
I20240503 08:12:19.091758 26133 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 327 FOLLOWER]: Advancing to term 329
I20240503 08:12:19.091765 26284 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 328 FOLLOWER]: Advancing to term 329
I20240503 08:12:19.092473 26284 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 329 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 329.
I20240503 08:12:19.092473 26133 raft_consensus.cc:2454] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb2b6702b7f495b92a4d8643d97b58d in term 329.
I20240503 08:12:19.092729 26176 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 329 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48; no voters:
I20240503 08:12:19.092834 26612 raft_consensus.cc:2781] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Leader election won for term 329
I20240503 08:12:19.092893 26612 raft_consensus.cc:686] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 LEADER]: Becoming Leader. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Running, Role: LEADER
I20240503 08:12:19.092933 26612 consensus_queue.cc:237] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14383, Committed index: 14383, Last appended: 326.14383, Last appended by leader: 14383, Current term: 329, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.093454 26028 catalog_manager.cc:5484] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d reported cstate change: term changed from 326 to 329, leader changed from db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193) to 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194). New cstate: current_term: 329 leader_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:19.121057 26133 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 326 index: 14383. Preceding OpId from leader: term: 329 index: 14384. (index mismatch)
I20240503 08:12:19.121416 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14384, Last known committed idx: 14383, Time since last communication: 0.000s
I20240503 08:12:19.125581 26284 raft_consensus.cc:1261] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 329 FOLLOWER]: Refusing update from remote peer 7fb2b6702b7f495b92a4d8643d97b58d: Log matching property violated. Preceding OpId in replica: term: 326 index: 14383. Preceding OpId from leader: term: 329 index: 14384. (index mismatch)
I20240503 08:12:19.125815 26612 consensus_queue.cc:1035] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14384, Last known committed idx: 14383, Time since last communication: 0.000s
I20240503 08:12:19.156229 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 329 FOLLOWER]: Starting leader election (detected failure of leader 7fb2b6702b7f495b92a4d8643d97b58d)
I20240503 08:12:19.156344 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 329 FOLLOWER]: Advancing to term 330
I20240503 08:12:19.157027 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 330 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.157160 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 330 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 329. Current term is 330. Ops: []
I20240503 08:12:19.157202 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 330 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
I20240503 08:12:19.157420 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 330 candidate_status { last_received { term: 329 index: 14384 } } ignore_live_leader: false dest_uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
I20240503 08:12:19.157382 26133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba72841b27b4d44b8f91e77a6d9f162" candidate_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" candidate_term: 330 candidate_status { last_received { term: 329 index: 14384 } } ignore_live_leader: false dest_uuid: "db1737abb1d5409bb049fa1c94d3ed48"
I20240503 08:12:19.157397 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 329.14384, Next index: 14385, Last known committed idx: 14384, Time since last communication: 0.000s
I20240503 08:12:19.157675 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 330 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:19.157755 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 330 FOLLOWER]: Leader election lost for term 330. Reason: could not achieve majority
W20240503 08:12:19.174211 26349 lock_manager.cc:494] Txn 9 has not acquired the partition lock after 1000ms
W20240503 08:12:19.174293 26348 lock_manager.cc:494] Txn 9 has not acquired the partition lock after 1000ms
I20240503 08:12:19.174669 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:12:19.177260 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:19.177359 17235 tablet_replica.cc:330] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: stopping tablet replica
I20240503 08:12:19.177423 17235 raft_consensus.cc:2229] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:19.177459 17235 raft_consensus.cc:2258] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:19.190387 26284 raft_consensus.cc:1226] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 330 FOLLOWER]: Rejecting Update request from peer 7fb2b6702b7f495b92a4d8643d97b58d for earlier term 329. Current term is 330. Ops: []
I20240503 08:12:19.190690 26612 consensus_queue.cc:1046] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 }, Status: INVALID_TERM, Last received: 329.14384, Next index: 14385, Last known committed idx: 14384, Time since last communication: 0.000s
I20240503 08:12:19.196110 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:12:19.196435 26627 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 330 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:19.196494 26627 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 330 FOLLOWER]: Advancing to term 331
I20240503 08:12:19.197305 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:12:19.197320 26627 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:19.197726 26627 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 331 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:19.197960 26249 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:12:19.198386 26249 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 331 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 08:12:19.198454 26251 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 331 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
I20240503 08:12:19.198529 26251 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 331 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:19.198668 26627 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Leader election lost for term 331. Reason: could not achieve majority
I20240503 08:12:19.200743 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:19.200874 17235 tablet_replica.cc:330] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: stopping tablet replica
I20240503 08:12:19.200942 17235 raft_consensus.cc:2229] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 LEADER]: Raft consensus shutting down.
W20240503 08:12:19.216392 26611 raft_consensus.cc:918] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 LEADER]: Unable to handle notification of new term (330): Illegal state: RaftConsensus is not running: State = Stopping
I20240503 08:12:19.216578 17235 raft_consensus.cc:2258] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:19.221419 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:12:19.222291 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:12:19.225595 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:19.225705 17235 tablet_replica.cc:330] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: stopping tablet replica
I20240503 08:12:19.225765 17235 raft_consensus.cc:2229] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:19.225805 17235 raft_consensus.cc:2258] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:19.251412 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:12:19.253028 17235 master.cc:537] Master@127.16.212.254:40039 shutting down...
I20240503 08:12:19.255544 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:19.255628 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:19.255648 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: stopping tablet replica
I20240503 08:12:19.267820 17235 master.cc:552] Master@127.16.212.254:40039 shutdown complete.
I20240503 08:12:19.269243 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:12:19.271733 17235 server_base.cc:1034] running on GCE node
W20240503 08:12:19.272326 26642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:19.272917 26645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:19.273473 26643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.273625 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:19.273661 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:19.273674 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723939273675 us; error 0 us; skew 500 ppm
I20240503 08:12:19.279623 17235 webserver.cc:460] Webserver started at http://127.16.212.254:44843/ using document root <none> and password file <none>
I20240503 08:12:19.279711 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:19.279740 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:12:19.280400 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.280833 26650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.280972 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.281030 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "b3cc48360c95405c90eb4737e611be95"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:12:19.281080 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:19.295822 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:19.296010 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:19.298688 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:40039
I20240503 08:12:19.302724 26702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:40039 every 8 connection(s)
I20240503 08:12:19.303407 26703 sys_catalog.cc:263] Verifying existing consensus state
I20240503 08:12:19.305513 26703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Bootstrap starting.
I20240503 08:12:19.313475 26703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Bootstrap replayed 1/1 log segments. Stats: ops{read=155 overwritten=0 applied=155 ignored=0} inserts{seen=5 ignored=0} mutations{seen=150 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:19.313766 26703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Bootstrap complete.
I20240503 08:12:19.313966 26703 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:12:19.314028 26703 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cc48360c95405c90eb4737e611be95, State: Initialized, Role: FOLLOWER
I20240503 08:12:19.314096 26703 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:12:19.314129 26703 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:19.314153 26703 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:19.314185 26703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:19.314791 26703 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:12:19.314848 26703 leader_election.cc:302] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3cc48360c95405c90eb4737e611be95; no voters:
I20240503 08:12:19.314960 26703 leader_election.cc:288] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 08:12:19.315016 26706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:19.315135 26703 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:19.315179 26706 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 2 LEADER]: Becoming Leader. State: Replica: b3cc48360c95405c90eb4737e611be95, State: Running, Role: LEADER
I20240503 08:12:19.315235 26706 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER }
I20240503 08:12:19.315439 26708 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3cc48360c95405c90eb4737e611be95. Latest consensus state: current_term: 2 leader_uuid: "b3cc48360c95405c90eb4737e611be95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER } }
I20240503 08:12:19.315539 26708 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:19.315452 26707 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b3cc48360c95405c90eb4737e611be95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cc48360c95405c90eb4737e611be95" member_type: VOTER } }
I20240503 08:12:19.315611 26707 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:19.315752 26712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:19.316061 26712 catalog_manager.cc:669] Loaded metadata for table test-workload [id=eb268649e74041bd8a290edbd04ed5a7]
I20240503 08:12:19.316280 26712 tablet_loader.cc:96] loaded metadata for tablet 5ba72841b27b4d44b8f91e77a6d9f162 (table test-workload [id=eb268649e74041bd8a290edbd04ed5a7])
I20240503 08:12:19.316327 26712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:19.316440 26712 catalog_manager.cc:1259] Loaded cluster ID: 4bb74111b3f64356abf9aba4b8e25d0d
I20240503 08:12:19.316459 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:19.316469 26712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:19.317307 26712 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:19.317442 26712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95: Loaded TSK: 0
I20240503 08:12:19.317516 26712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:19.319938 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:19.322149 26723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:19.322681 26722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.323241 17235 server_base.cc:1034] running on GCE node
W20240503 08:12:19.323854 26725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.324028 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:19.324103 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:19.324119 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723939324120 us; error 0 us; skew 500 ppm
I20240503 08:12:19.324887 17235 webserver.cc:460] Webserver started at http://127.16.212.193:44983/ using document root <none> and password file <none>
I20240503 08:12:19.324975 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:19.325011 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:12:19.325593 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:12:19.326174 26730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.326342 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:12:19.326400 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "db1737abb1d5409bb049fa1c94d3ed48"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:12:19.326462 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:19.331629 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:19.331830 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:19.331966 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:19.332245 26737 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 08:12:19.333096 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:12:19.333145 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:19.333184 17235 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 08:12:19.333608 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:12:19.333638 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.333658 26737 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Bootstrap starting.
I20240503 08:12:19.337142 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:40937
I20240503 08:12:19.337450 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:19.345181 26798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.345366 26795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:40937 every 8 connection(s)
W20240503 08:12:19.347395 26801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:19.347972 26799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.349056 26796 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40039
I20240503 08:12:19.349141 26796 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:19.349257 26796 heartbeater.cc:511] Master 127.16.212.254:40039 requested a full tablet report, sending...
I20240503 08:12:19.349402 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:19.349524 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:19.349552 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:19.349570 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723939349569 us; error 0 us; skew 500 ppm
I20240503 08:12:19.349581 26667 ts_manager.cc:194] Registered new tserver with Master: db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:40937)
I20240503 08:12:19.350381 26667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47548
I20240503 08:12:19.350548 17235 webserver.cc:460] Webserver started at http://127.16.212.194:42783/ using document root <none> and password file <none>
I20240503 08:12:19.350661 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:19.350698 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:12:19.351262 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240503 08:12:19.351755 26809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.351862 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.351902 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "7fb2b6702b7f495b92a4d8643d97b58d"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:12:19.351958 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:19.357275 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:19.357527 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:19.357653 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:19.358429 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:12:19.358479 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:19.358938 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:12:19.358980 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.359047 26816 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap starting.
I20240503 08:12:19.362996 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:42113
I20240503 08:12:19.363330 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:19.365171 26878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.370771 26874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:42113 every 8 connection(s)
I20240503 08:12:19.373703 17235 server_base.cc:1034] running on GCE node
W20240503 08:12:19.374200 26881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:19.374423 26879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:19.374703 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:19.374742 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:19.374764 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723939374763 us; error 0 us; skew 500 ppm
I20240503 08:12:19.374887 26875 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40039
I20240503 08:12:19.374954 26875 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:19.375082 26875 heartbeater.cc:511] Master 127.16.212.254:40039 requested a full tablet report, sending...
I20240503 08:12:19.375356 26667 ts_manager.cc:194] Registered new tserver with Master: 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:42113)
I20240503 08:12:19.375641 17235 webserver.cc:460] Webserver started at http://127.16.212.195:37291/ using document root <none> and password file <none>
I20240503 08:12:19.375730 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:19.375762 17235 fs_manager.cc:362] Using existing metadata directory in first data directory
I20240503 08:12:19.375896 26667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47558
I20240503 08:12:19.376446 17235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:19.377035 26886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.377285 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:19.377385 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "e8eb2ae79c07480d86c3ecf1261153aa"
format_stamp: "Formatted at 2024-05-03 08:11:52 on dist-test-slave-zl2t"
I20240503 08:12:19.377503 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:19.410343 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:19.410737 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:19.411068 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:19.411927 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 08:12:19.412047 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:19.412685 17235 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 08:12:19.412717 26893 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Bootstrap starting.
I20240503 08:12:19.412781 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:19.420833 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:38841
I20240503 08:12:19.431253 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:38841 every 8 connection(s)
I20240503 08:12:19.435240 26952 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40039
I20240503 08:12:19.435344 26952 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:19.435475 26952 heartbeater.cc:511] Master 127.16.212.254:40039 requested a full tablet report, sending...
I20240503 08:12:19.435776 26667 ts_manager.cc:194] Registered new tserver with Master: e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:38841)
I20240503 08:12:19.436282 26667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47570
I20240503 08:12:19.436398 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015367426s
I20240503 08:12:20.111375 26893 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Bootstrap replayed 1/3 log segments. Stats: ops{read=4968 overwritten=0 applied=4967 ignored=0} inserts{seen=248300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:20.351009 26796 heartbeater.cc:503] Master 127.16.212.254:40039 was elected leader, sending a full tablet report...
I20240503 08:12:20.376562 26875 heartbeater.cc:503] Master 127.16.212.254:40039 was elected leader, sending a full tablet report...
I20240503 08:12:20.437518 26952 heartbeater.cc:503] Master 127.16.212.254:40039 was elected leader, sending a full tablet report...
I20240503 08:12:20.514695 26816 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap replayed 1/4 log segments. Stats: ops{read=4631 overwritten=0 applied=4630 ignored=0} inserts{seen=231450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:20.553599 26737 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Bootstrap replayed 1/3 log segments. Stats: ops{read=4988 overwritten=0 applied=4985 ignored=0} inserts{seen=249200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:12:21.085906 26893 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Bootstrap replayed 2/3 log segments. Stats: ops{read=9932 overwritten=0 applied=9929 ignored=0} inserts{seen=496400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:12:21.255056 26737 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Bootstrap replayed 2/3 log segments. Stats: ops{read=9958 overwritten=0 applied=9957 ignored=0} inserts{seen=497800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 08:12:21.694566 26816 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap replayed 2/4 log segments. Stats: ops{read=9262 overwritten=0 applied=9259 ignored=0} inserts{seen=462900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:12:21.874599 26737 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Bootstrap replayed 3/3 log segments. Stats: ops{read=14393 overwritten=9 applied=14384 ignored=0} inserts{seen=711602 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:21.875054 26737 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Bootstrap complete.
I20240503 08:12:21.880025 26737 ts_tablet_manager.cc:1367] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Time spent bootstrapping tablet: real 2.546s user 2.110s sys 0.428s
I20240503 08:12:21.880203 26737 raft_consensus.cc:348] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:21.880280 26737 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1737abb1d5409bb049fa1c94d3ed48, State: Initialized, Role: FOLLOWER
I20240503 08:12:21.880331 26737 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14384, Last appended: 329.14384, Last appended by leader: 14384, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:21.880534 26737 ts_tablet_manager.cc:1398] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20240503 08:12:21.905776 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.905865 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 329 FOLLOWER]: Advancing to term 330
I20240503 08:12:21.906581 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 330 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:21.907312 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 330 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:21.907550 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 330 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
W20240503 08:12:21.907771 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 330 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
I20240503 08:12:21.907804 26731 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 330 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:21.907866 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 330 FOLLOWER]: Leader election lost for term 330. Reason: could not achieve majority
I20240503 08:12:21.947778 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 330 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:21.947867 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 330 FOLLOWER]: Advancing to term 331
I20240503 08:12:21.948546 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 331 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:21.948879 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 331 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:21.949337 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 331 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
W20240503 08:12:21.949412 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 331 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:21.949445 26732 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 331 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:21.949539 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 331 FOLLOWER]: Leader election lost for term 331. Reason: could not achieve majority
I20240503 08:12:22.014416 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 331 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.014523 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 331 FOLLOWER]: Advancing to term 332
I20240503 08:12:22.015434 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 332 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.015614 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 332 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.016175 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 332 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
W20240503 08:12:22.016227 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 332 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.016300 26732 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 332 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:22.016443 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 332 FOLLOWER]: Leader election lost for term 332. Reason: could not achieve majority
I20240503 08:12:22.095681 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 332 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.095798 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 332 FOLLOWER]: Advancing to term 333
I20240503 08:12:22.096691 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 333 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.097249 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 333 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.097455 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 333 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
W20240503 08:12:22.097684 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 333 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.097751 26732 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 333 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:22.097831 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 333 FOLLOWER]: Leader election lost for term 333. Reason: could not achieve majority
I20240503 08:12:22.116199 26893 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Bootstrap replayed 3/3 log segments. Stats: ops{read=14391 overwritten=7 applied=14384 ignored=0} inserts{seen=711602 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:22.116639 26893 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Bootstrap complete.
I20240503 08:12:22.121479 26893 ts_tablet_manager.cc:1367] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Time spent bootstrapping tablet: real 2.709s user 2.220s sys 0.480s
I20240503 08:12:22.121636 26893 raft_consensus.cc:348] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.121708 26893 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8eb2ae79c07480d86c3ecf1261153aa, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.121752 26893 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14384, Last appended: 329.14384, Last appended by leader: 14384, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.121960 26893 ts_tablet_manager.cc:1398] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa: Time spent starting tablet: real 0.000s user 0.002s sys 0.001s
I20240503 08:12:22.150385 26968 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.150496 26968 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 331 FOLLOWER]: Advancing to term 332
I20240503 08:12:22.151455 26968 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 332 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.151677 26968 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 332 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.152484 26890 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 332 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
W20240503 08:12:22.152619 26888 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 332 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.152665 26888 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 332 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:22.152735 26968 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 332 FOLLOWER]: Leader election lost for term 332. Reason: could not achieve majority
I20240503 08:12:22.191504 26968 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 332 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.191596 26968 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 332 FOLLOWER]: Advancing to term 333
I20240503 08:12:22.192284 26968 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 333 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.192612 26968 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 333 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.193010 26890 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 333 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
W20240503 08:12:22.193244 26888 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 333 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.193290 26888 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 333 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:22.193375 26968 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 333 FOLLOWER]: Leader election lost for term 333. Reason: could not achieve majority
I20240503 08:12:22.203387 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 333 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.203473 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 333 FOLLOWER]: Advancing to term 334
I20240503 08:12:22.204111 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 334 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.204267 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 334 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.204849 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 334 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
W20240503 08:12:22.204902 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 334 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.204970 26731 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 334 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:22.205058 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 334 FOLLOWER]: Leader election lost for term 334. Reason: could not achieve majority
I20240503 08:12:22.259209 26968 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 333 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.259326 26968 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 333 FOLLOWER]: Advancing to term 334
I20240503 08:12:22.260216 26968 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 334 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.260390 26968 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 334 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.260969 26890 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 334 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
W20240503 08:12:22.261065 26888 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 334 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.261086 26888 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 334 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:22.261176 26968 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 334 FOLLOWER]: Leader election lost for term 334. Reason: could not achieve majority
I20240503 08:12:22.291899 26968 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 334 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.291996 26968 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 334 FOLLOWER]: Advancing to term 335
I20240503 08:12:22.292644 26968 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 335 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.292796 26968 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 335 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.293391 26888 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 335 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
W20240503 08:12:22.293460 26890 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 335 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
I20240503 08:12:22.293519 26890 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 335 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:22.293632 26968 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 335 FOLLOWER]: Leader election lost for term 335. Reason: could not achieve majority
I20240503 08:12:22.351835 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 334 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.351948 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 334 FOLLOWER]: Advancing to term 335
I20240503 08:12:22.352885 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 335 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.353077 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 335 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.353612 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 335 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
W20240503 08:12:22.353672 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 335 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.353727 26731 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 335 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:22.353789 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 335 FOLLOWER]: Leader election lost for term 335. Reason: could not achieve majority
I20240503 08:12:22.390380 26963 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 335 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.390484 26963 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 335 FOLLOWER]: Advancing to term 336
I20240503 08:12:22.391201 26963 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 336 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.391356 26963 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 336 election: Requested vote from peers e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.391909 26732 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 336 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
W20240503 08:12:22.391983 26731 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 336 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
I20240503 08:12:22.392015 26731 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [CANDIDATE]: Term 336 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1737abb1d5409bb049fa1c94d3ed48; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:22.392134 26963 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 336 FOLLOWER]: Leader election lost for term 336. Reason: could not achieve majority
I20240503 08:12:22.392979 26968 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 335 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.393025 26968 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 335 FOLLOWER]: Advancing to term 336
I20240503 08:12:22.393518 26968 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 336 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.393736 26968 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 336 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.394040 26890 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 336 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
W20240503 08:12:22.394233 26888 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 336 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.394276 26888 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 336 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:22.394358 26968 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 336 FOLLOWER]: Leader election lost for term 336. Reason: could not achieve majority
I20240503 08:12:22.503360 26816 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap replayed 3/4 log segments. Stats: ops{read=13892 overwritten=0 applied=13889 ignored=0} inserts{seen=694400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20240503 08:12:22.525869 26968 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 336 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.525967 26968 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 336 FOLLOWER]: Advancing to term 337
I20240503 08:12:22.526707 26968 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 337 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.527117 26968 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 337 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443)
W20240503 08:12:22.527431 26890 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 337 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
W20240503 08:12:22.527491 26888 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 337 election: RPC error from VoteRequest() call to peer 7fb2b6702b7f495b92a4d8643d97b58d (127.16.212.194:40443): Network error: Client connection negotiation failed: client connection to 127.16.212.194:40443: connect: Connection refused (error 111)
I20240503 08:12:22.527551 26888 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [CANDIDATE]: Term 337 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8eb2ae79c07480d86c3ecf1261153aa; no voters: 7fb2b6702b7f495b92a4d8643d97b58d, db1737abb1d5409bb049fa1c94d3ed48
I20240503 08:12:22.527653 26968 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 337 FOLLOWER]: Leader election lost for term 337. Reason: could not achieve majority
I20240503 08:12:22.556941 26816 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap replayed 4/4 log segments. Stats: ops{read=14388 overwritten=4 applied=14384 ignored=0} inserts{seen=711602 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:22.557390 26816 tablet_bootstrap.cc:492] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Bootstrap complete.
I20240503 08:12:22.562548 26816 ts_tablet_manager.cc:1367] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Time spent bootstrapping tablet: real 3.204s user 2.705s sys 0.484s
I20240503 08:12:22.562747 26816 raft_consensus.cc:348] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.562819 26816 raft_consensus.cc:729] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb2b6702b7f495b92a4d8643d97b58d, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.562880 26816 consensus_queue.cc:260] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14384, Last appended: 329.14384, Last appended by leader: 14384, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.563042 26816 ts_tablet_manager.cc:1398] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.563228 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:12:22.565596 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:22.565696 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:12:22.565748 17235 raft_consensus.cc:2229] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 336 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:22.565789 17235 raft_consensus.cc:2258] T 5ba72841b27b4d44b8f91e77a6d9f162 P db1737abb1d5409bb049fa1c94d3ed48 [term 336 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:22.583988 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:12:22.584966 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:12:22.585373 26973 raft_consensus.cc:482] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.585448 26973 raft_consensus.cc:3037] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 329 FOLLOWER]: Advancing to term 330
I20240503 08:12:22.586248 26973 raft_consensus.cc:504] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 330 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1737abb1d5409bb049fa1c94d3ed48" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 33927 } } peers { permanent_uuid: "e8eb2ae79c07480d86c3ecf1261153aa" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 44771 } } peers { permanent_uuid: "7fb2b6702b7f495b92a4d8643d97b58d" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 40443 } }
I20240503 08:12:22.586709 26973 leader_election.cc:288] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 330 election: Requested vote from peers db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927), e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771)
I20240503 08:12:22.587702 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
W20240503 08:12:22.587745 26813 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 330 election: RPC error from VoteRequest() call to peer db1737abb1d5409bb049fa1c94d3ed48 (127.16.212.193:33927): Network error: Client connection negotiation failed: client connection to 127.16.212.193:33927: connect: Connection refused (error 111)
W20240503 08:12:22.587813 26810 leader_election.cc:334] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 330 election: RPC error from VoteRequest() call to peer e8eb2ae79c07480d86c3ecf1261153aa (127.16.212.195:44771): Network error: Client connection negotiation failed: client connection to 127.16.212.195:44771: connect: Connection refused (error 111)
I20240503 08:12:22.587860 26810 leader_election.cc:302] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [CANDIDATE]: Term 330 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fb2b6702b7f495b92a4d8643d97b58d; no voters: db1737abb1d5409bb049fa1c94d3ed48, e8eb2ae79c07480d86c3ecf1261153aa
I20240503 08:12:22.587934 26973 raft_consensus.cc:2726] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 330 FOLLOWER]: Leader election lost for term 330. Reason: could not achieve majority
I20240503 08:12:22.588012 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:12:22.588073 17235 raft_consensus.cc:2229] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 330 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:22.588130 17235 raft_consensus.cc:2258] T 5ba72841b27b4d44b8f91e77a6d9f162 P 7fb2b6702b7f495b92a4d8643d97b58d [term 330 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:22.601634 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:12:22.602603 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:12:22.605110 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:22.605285 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:12:22.605332 17235 raft_consensus.cc:2229] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 337 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:22.605391 17235 raft_consensus.cc:2258] T 5ba72841b27b4d44b8f91e77a6d9f162 P e8eb2ae79c07480d86c3ecf1261153aa [term 337 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:22.628898 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:12:22.629858 17235 master.cc:537] Master@127.16.212.254:40039 shutting down...
I20240503 08:12:22.631894 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:12:22.631965 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3cc48360c95405c90eb4737e611be95 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:22.631985 17235 tablet_replica.cc:330] stopping tablet replica
I20240503 08:12:22.643364 17235 master.cc:552] Master@127.16.212.254:40039 shutdown complete.
[ OK ] TxnParticipantElectionStormITest.TestFrequentElections (30332 ms)
[ RUN ] TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm
I20240503 08:12:22.664415 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:39025
I20240503 08:12:22.664602 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:22.666766 26978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.667321 26977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.668010 26980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:22.668606 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:22.668741 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:22.668771 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:22.668793 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723942668793 us; error 0 us; skew 500 ppm
I20240503 08:12:22.669472 17235 webserver.cc:460] Webserver started at http://127.16.212.254:37073/ using document root <none> and password file <none>
I20240503 08:12:22.669565 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:22.669620 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:22.669679 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:22.669966 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "6281f4184fd942d09017a4d9c30bf205"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.670931 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:22.671629 26985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.671878 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:22.671936 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "6281f4184fd942d09017a4d9c30bf205"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.671993 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:22.684260 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:22.684484 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:22.687254 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:39025
I20240503 08:12:22.689934 27038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.690085 27037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:39025 every 8 connection(s)
I20240503 08:12:22.693202 27038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205: Bootstrap starting.
I20240503 08:12:22.693518 27038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.694151 27038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205: No bootstrap required, opened a new log
I20240503 08:12:22.694260 27038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f4184fd942d09017a4d9c30bf205" member_type: VOTER }
I20240503 08:12:22.694324 27038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.694339 27038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6281f4184fd942d09017a4d9c30bf205, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.694389 27038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f4184fd942d09017a4d9c30bf205" member_type: VOTER }
I20240503 08:12:22.694425 27038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:22.694451 27038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:22.694479 27038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.695020 27038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f4184fd942d09017a4d9c30bf205" member_type: VOTER }
I20240503 08:12:22.695076 27038 leader_election.cc:302] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6281f4184fd942d09017a4d9c30bf205; no voters:
I20240503 08:12:22.695185 27038 leader_election.cc:288] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:22.695264 27041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:22.695340 27038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:22.695405 27041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 1 LEADER]: Becoming Leader. State: Replica: 6281f4184fd942d09017a4d9c30bf205, State: Running, Role: LEADER
I20240503 08:12:22.695456 27041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f4184fd942d09017a4d9c30bf205" member_type: VOTER }
I20240503 08:12:22.695669 27042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6281f4184fd942d09017a4d9c30bf205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f4184fd942d09017a4d9c30bf205" member_type: VOTER } }
I20240503 08:12:22.695686 27043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6281f4184fd942d09017a4d9c30bf205. Latest consensus state: current_term: 1 leader_uuid: "6281f4184fd942d09017a4d9c30bf205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6281f4184fd942d09017a4d9c30bf205" member_type: VOTER } }
I20240503 08:12:22.695755 27042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:22.695761 27043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:22.696033 27046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:22.696197 27046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:22.696727 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:22.696745 27046 catalog_manager.cc:1347] Generated new cluster ID: e3fb3eddf2864343989113ff88134bfa
I20240503 08:12:22.696800 27046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:22.704898 27046 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:22.705363 27046 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:22.712139 27046 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205: Generated new TSK 0
I20240503 08:12:22.712224 27046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:22.728655 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:22.730722 27060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.731263 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.731998 27062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:22.732672 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:22.732769 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:22.732807 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:22.732839 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723942732839 us; error 0 us; skew 500 ppm
I20240503 08:12:22.733546 17235 webserver.cc:460] Webserver started at http://127.16.212.193:43461/ using document root <none> and password file <none>
I20240503 08:12:22.733654 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:22.733709 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:22.733772 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:22.734083 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "42e3d7884bb84e129ded61a8271632af"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.735025 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:22.735486 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.735627 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.735670 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "42e3d7884bb84e129ded61a8271632af"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.735711 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:22.751552 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:22.751778 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:22.751926 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:22.752676 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:22.752718 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.752759 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:22.752784 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.755560 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:41641
I20240503 08:12:22.756449 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 08:12:22.756572 27130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:41641 every 8 connection(s)
W20240503 08:12:22.758460 27133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.758945 27134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.759626 27136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:22.760414 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:22.760460 27131 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39025
I20240503 08:12:22.760540 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 08:12:22.760562 27131 heartbeater.cc:464] Registering TS with master...
W20240503 08:12:22.760571 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:22.760622 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723942760622 us; error 0 us; skew 500 ppm
I20240503 08:12:22.760695 27131 heartbeater.cc:511] Master 127.16.212.254:39025 requested a full tablet report, sending...
I20240503 08:12:22.760967 27002 ts_manager.cc:194] Registered new tserver with Master: 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641)
I20240503 08:12:22.761416 17235 webserver.cc:460] Webserver started at http://127.16.212.194:42503/ using document root <none> and password file <none>
I20240503 08:12:22.761515 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:22.761533 27002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41080
I20240503 08:12:22.761569 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:22.761639 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:22.761976 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "5a436c985aca40adbd15dbee459a01ca"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.763015 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:22.763561 27143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.763708 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:12:22.763761 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "5a436c985aca40adbd15dbee459a01ca"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.763803 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:22.777027 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:22.777220 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:22.777364 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:22.777523 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:22.777546 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.777568 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:22.777586 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.780434 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:45047
I20240503 08:12:22.780509 27206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:45047 every 8 connection(s)
I20240503 08:12:22.780787 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:22.782804 27210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:22.783520 27211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:22.784001 27207 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39025
I20240503 08:12:22.784075 27207 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:22.784197 27207 heartbeater.cc:511] Master 127.16.212.254:39025 requested a full tablet report, sending...
W20240503 08:12:22.784430 27213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:22.784488 27002 ts_manager.cc:194] Registered new tserver with Master: 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:22.784868 27002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41084
I20240503 08:12:22.785039 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:22.785149 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:22.785187 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:22.785207 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723942785206 us; error 0 us; skew 500 ppm
I20240503 08:12:22.785887 17235 webserver.cc:460] Webserver started at http://127.16.212.195:41019/ using document root <none> and password file <none>
I20240503 08:12:22.785956 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:22.786012 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:22.786047 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:22.786305 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.787276 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:22.787863 27218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.788033 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240503 08:12:22.788085 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
format_stamp: "Formatted at 2024-05-03 08:12:22 on dist-test-slave-zl2t"
I20240503 08:12:22.788139 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:22.807678 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:22.807932 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:22.808101 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:22.808295 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:22.808322 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.808357 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:22.808382 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:22.811585 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:45571
I20240503 08:12:22.811654 27281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:45571 every 8 connection(s)
I20240503 08:12:22.814695 27282 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39025
I20240503 08:12:22.814766 27282 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:22.814872 27282 heartbeater.cc:511] Master 127.16.212.254:39025 requested a full tablet report, sending...
I20240503 08:12:22.815158 27002 ts_manager.cc:194] Registered new tserver with Master: d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:22.815491 27002 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41094
I20240503 08:12:22.816162 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004411351s
I20240503 08:12:22.816526 17235 test_util.cc:268] Using random seed: 1199561485
I20240503 08:12:22.821323 27002 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41108:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:12:22.821473 27002 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:12:22.825860 27095 tablet_service.cc:1467] Processing CreateTablet for tablet dfda1db397144737b1311452c50846fd (DEFAULT_TABLE table=test-workload [id=a69e458618c844bc8c0afe45ea858a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.826026 27095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfda1db397144737b1311452c50846fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.826488 27171 tablet_service.cc:1467] Processing CreateTablet for tablet dfda1db397144737b1311452c50846fd (DEFAULT_TABLE table=test-workload [id=a69e458618c844bc8c0afe45ea858a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.826596 27246 tablet_service.cc:1467] Processing CreateTablet for tablet dfda1db397144737b1311452c50846fd (DEFAULT_TABLE table=test-workload [id=a69e458618c844bc8c0afe45ea858a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:22.826625 27171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfda1db397144737b1311452c50846fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.826774 27246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfda1db397144737b1311452c50846fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:22.827852 27297 tablet_bootstrap.cc:492] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: Bootstrap starting.
I20240503 08:12:22.828034 27298 tablet_bootstrap.cc:492] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Bootstrap starting.
I20240503 08:12:22.828191 27299 tablet_bootstrap.cc:492] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: Bootstrap starting.
I20240503 08:12:22.828418 27297 tablet_bootstrap.cc:654] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.828604 27298 tablet_bootstrap.cc:654] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.828685 27299 tablet_bootstrap.cc:654] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:22.829001 27297 tablet_bootstrap.cc:492] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: No bootstrap required, opened a new log
I20240503 08:12:22.829071 27297 ts_tablet_manager.cc:1367] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:22.829221 27298 tablet_bootstrap.cc:492] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: No bootstrap required, opened a new log
I20240503 08:12:22.829263 27298 ts_tablet_manager.cc:1367] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:22.829236 27297 raft_consensus.cc:348] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.829313 27297 raft_consensus.cc:374] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.829344 27297 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e3d7884bb84e129ded61a8271632af, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.829391 27298 raft_consensus.cc:348] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.829397 27297 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.829470 27298 raft_consensus.cc:374] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.829490 27298 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.829540 27298 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.829622 27131 heartbeater.cc:503] Master 127.16.212.254:39025 was elected leader, sending a full tablet report...
I20240503 08:12:22.829692 27298 ts_tablet_manager.cc:1398] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:12:22.829730 27207 heartbeater.cc:503] Master 127.16.212.254:39025 was elected leader, sending a full tablet report...
I20240503 08:12:22.829788 27297 ts_tablet_manager.cc:1398] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:22.830149 27299 tablet_bootstrap.cc:492] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: No bootstrap required, opened a new log
I20240503 08:12:22.830210 27299 ts_tablet_manager.cc:1367] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:22.830365 27299 raft_consensus.cc:348] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.830428 27299 raft_consensus.cc:374] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:22.830448 27299 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2620a06bcb6443f8bd1f4f48bc4e0f8, State: Initialized, Role: FOLLOWER
I20240503 08:12:22.830488 27299 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.830663 27299 ts_tablet_manager.cc:1398] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20240503 08:12:22.830678 27282 heartbeater.cc:503] Master 127.16.212.254:39025 was elected leader, sending a full tablet report...
I20240503 08:12:22.846876 27303 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:22.846989 27303 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.847175 27303 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:22.848647 27304 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:22.848726 27304 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.848883 27304 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:22.848974 27305 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:22.849063 27305 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.849857 27305 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:22.850214 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca" is_pre_election: true
I20240503 08:12:22.850302 27181 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e3d7884bb84e129ded61a8271632af in term 0.
I20240503 08:12:22.850469 27071 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca; no voters:
I20240503 08:12:22.850724 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" is_pre_election: true
I20240503 08:12:22.850790 27256 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e3d7884bb84e129ded61a8271632af in term 0.
I20240503 08:12:22.852929 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" is_pre_election: true
I20240503 08:12:22.853022 27256 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a436c985aca40adbd15dbee459a01ca in term 0.
I20240503 08:12:22.853161 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters:
I20240503 08:12:22.853256 27304 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:22.853283 27304 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.853305 27304 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.853307 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca" is_pre_election: true
I20240503 08:12:22.853478 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af" is_pre_election: true
I20240503 08:12:22.853533 27105 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a436c985aca40adbd15dbee459a01ca in term 0.
I20240503 08:12:22.853600 27303 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:22.853623 27303 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:22.853641 27303 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.853688 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af" is_pre_election: true
I20240503 08:12:22.854194 27304 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.854303 27303 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.854337 27181 raft_consensus.cc:2379] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 in current term 1: Already voted for candidate 5a436c985aca40adbd15dbee459a01ca in this term.
I20240503 08:12:22.854360 27304 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 1 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:22.854522 27104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:22.854545 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:22.854624 27256 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:22.854640 27105 raft_consensus.cc:2379] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 in current term 1: Already voted for candidate 42e3d7884bb84e129ded61a8271632af in this term.
I20240503 08:12:22.854771 27221 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca
I20240503 08:12:22.854843 27303 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 1 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:22.854893 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:22.854919 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:22.854943 27181 raft_consensus.cc:2379] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e3d7884bb84e129ded61a8271632af in current term 1: Already voted for candidate 5a436c985aca40adbd15dbee459a01ca in this term.
I20240503 08:12:22.855089 27070 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af; no voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:22.855197 27303 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:12:22.855358 27256 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a436c985aca40adbd15dbee459a01ca in term 1.
I20240503 08:12:22.855427 27305 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 08:12:22.855492 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af
I20240503 08:12:22.855587 27304 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:22.855731 27304 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 LEADER]: Becoming Leader. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Running, Role: LEADER
I20240503 08:12:22.855775 27304 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:22.856300 27002 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a436c985aca40adbd15dbee459a01ca (127.16.212.194). New cstate: current_term: 1 leader_uuid: "5a436c985aca40adbd15dbee459a01ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:22.871269 27105 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 1 FOLLOWER]: Refusing update from remote peer 5a436c985aca40adbd15dbee459a01ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:22.871269 27256 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 1 FOLLOWER]: Refusing update from remote peer 5a436c985aca40adbd15dbee459a01ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:22.871650 27312 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:22.871776 27304 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:32.895552 27355 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a436c985aca40adbd15dbee459a01ca)
I20240503 08:12:32.895678 27355 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:32.896591 27355 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:32.896759 27355 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 2 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:32.896986 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 2 candidate_status { last_received { term: 1 index: 13764 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:32.897006 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 2 candidate_status { last_received { term: 1 index: 13764 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:32.897322 27071 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af; no voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:32.897460 27355 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 08:12:32.899786 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 2 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 1. Current term is 2. Ops: []
I20240503 08:12:32.900050 27328 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 1.13764, Next index: 13765, Last known committed idx: 13764, Time since last communication: 0.000s
I20240503 08:12:32.919287 27328 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:32.919379 27328 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Running, Role: LEADER
I20240503 08:12:32.919440 27328 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13764, Committed index: 13764, Last appended: 1.13764, Last appended by leader: 13764, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:32.919559 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:32.933594 27355 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:32.933709 27355 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:32.934393 27355 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:32.934552 27355 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 3 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:32.934844 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 3 candidate_status { last_received { term: 1 index: 13764 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:32.934859 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 3 candidate_status { last_received { term: 1 index: 13764 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:32.934924 27181 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:32.935573 27181 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e3d7884bb84e129ded61a8271632af in term 3.
I20240503 08:12:32.935824 27071 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca; no voters: d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:32.935914 27355 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 3 FOLLOWER]: Leader election won for term 3
I20240503 08:12:32.936064 27355 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 3 LEADER]: Becoming Leader. State: Replica: 42e3d7884bb84e129ded61a8271632af, State: Running, Role: LEADER
I20240503 08:12:32.936116 27355 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13764, Committed index: 13764, Last appended: 1.13764, Last appended by leader: 13764, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:32.936556 26995 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af reported cstate change: term changed from 1 to 3, leader changed from 5a436c985aca40adbd15dbee459a01ca (127.16.212.194) to 42e3d7884bb84e129ded61a8271632af (127.16.212.193). New cstate: current_term: 3 leader_uuid: "42e3d7884bb84e129ded61a8271632af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:32.939287 27171 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60650: Illegal state: replica 5a436c985aca40adbd15dbee459a01ca is not leader of this config: current role FOLLOWER
I20240503 08:12:32.945624 27364 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a436c985aca40adbd15dbee459a01ca)
I20240503 08:12:32.945709 27364 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:32.946657 27364 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:32.946898 27364 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 2 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:32.946966 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 13764 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:32.947074 27181 raft_consensus.cc:2354] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 2. Current term is 3.
I20240503 08:12:32.946983 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 13764 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:32.947331 27256 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 08:12:32.947342 27222 leader_election.cc:398] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 2 election: Vote denied by peer 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047) with higher term. Message: Invalid argument: T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 2. Current term is 3.
I20240503 08:12:32.947413 27181 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Refusing update from remote peer 42e3d7884bb84e129ded61a8271632af: Log matching property violated. Preceding OpId in replica: term: 1 index: 13764. Preceding OpId from leader: term: 3 index: 13766. (index mismatch)
I20240503 08:12:32.947428 27222 leader_election.cc:401] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240503 08:12:32.947598 27355 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13765, Last known committed idx: 13764, Time since last communication: 0.000s
I20240503 08:12:32.948371 27256 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 3 FOLLOWER]: Refusing update from remote peer 42e3d7884bb84e129ded61a8271632af: Log matching property violated. Preceding OpId in replica: term: 1 index: 13764. Preceding OpId from leader: term: 3 index: 13766. (index mismatch)
I20240503 08:12:32.948567 27355 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13765, Last known committed idx: 13764, Time since last communication: 0.000s
I20240503 08:12:32.948659 27364 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047) with higher term. Message: Invalid argument: T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 2. Current term is 3.
I20240503 08:12:32.949831 27320 mvcc.cc:205] Tried to move back new op lower bound from 7023509311271063552 to 7023509311226871808. Current Snapshot: MvccSnapshot[applied={T|T < 7023509311271063552}]
I20240503 08:12:32.990288 27366 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42e3d7884bb84e129ded61a8271632af)
I20240503 08:12:32.990378 27366 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:32.991125 27366 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:32.991292 27366 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 4 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:32.991489 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 13767 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:32.991506 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 13767 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:32.991775 27222 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca
I20240503 08:12:32.991896 27366 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:12:32.992689 27256 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 3. Current term is 4. Ops: []
I20240503 08:12:32.992920 27365 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 3.13767, Next index: 13768, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.019117 27328 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42e3d7884bb84e129ded61a8271632af)
I20240503 08:12:33.019223 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:33.019924 27328 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.020082 27328 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 4 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.020347 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 4 candidate_status { last_received { term: 3 index: 13767 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.020351 27256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 4 candidate_status { last_received { term: 3 index: 13767 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.020459 27255 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 3. Current term is 4. Ops: []
I20240503 08:12:33.020660 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca; no voters: 42e3d7884bb84e129ded61a8271632af, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.020687 27356 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 3.13767, Next index: 13768, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.020823 27328 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 08:12:33.021229 27255 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 3. Current term is 4. Ops: [3.13768-3.13768]
I20240503 08:12:33.021258 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 4 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 3. Current term is 4. Ops: [3.13768-3.13768]
I20240503 08:12:33.021409 27356 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 3.13767, Next index: 13768, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.021490 27365 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: INVALID_TERM, Last received: 3.13767, Next index: 13768, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.025535 27355 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 3 LEADER]: Stepping down as leader of term 3
I20240503 08:12:33.025586 27355 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42e3d7884bb84e129ded61a8271632af, State: Running, Role: LEADER
I20240503 08:12:33.025635 27355 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13767, Committed index: 13767, Last appended: 3.13768, Last appended by leader: 13768, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.025748 27355 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 3 FOLLOWER]: Advancing to term 4
I20240503 08:12:33.028611 27366 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.028676 27366 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:33.029276 27366 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.029444 27366 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 5 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:33.029649 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 5 candidate_status { last_received { term: 3 index: 13767 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.029649 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 5 candidate_status { last_received { term: 3 index: 13767 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.029740 27105 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:33.029747 27181 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 4 FOLLOWER]: Advancing to term 5
I20240503 08:12:33.030345 27181 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 in term 5.
I20240503 08:12:33.030380 27105 raft_consensus.cc:2396] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 for term 5 because replica has last-logged OpId of term: 3 index: 13768, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 13767.
I20240503 08:12:33.030576 27222 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af
I20240503 08:12:33.030673 27366 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 08:12:33.030733 27366 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 5 LEADER]: Becoming Leader. State: Replica: d2620a06bcb6443f8bd1f4f48bc4e0f8, State: Running, Role: LEADER
I20240503 08:12:33.030786 27366 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13767, Committed index: 13767, Last appended: 3.13767, Last appended by leader: 13767, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.031296 26995 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 reported cstate change: term changed from 3 to 5, leader changed from 42e3d7884bb84e129ded61a8271632af (127.16.212.193) to d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195). New cstate: current_term: 5 leader_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:33.052683 27356 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.052811 27356 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:12:33.053578 27356 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.053751 27356 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 6 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.053974 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 6 candidate_status { last_received { term: 3 index: 13768 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.054049 27181 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:12:33.054028 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 6 candidate_status { last_received { term: 3 index: 13768 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.054687 27181 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e3d7884bb84e129ded61a8271632af in term 6.
I20240503 08:12:33.054878 27071 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca; no voters: d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.054998 27356 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 FOLLOWER]: Leader election won for term 6
I20240503 08:12:33.055055 27356 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 LEADER]: Becoming Leader. State: Replica: 42e3d7884bb84e129ded61a8271632af, State: Running, Role: LEADER
I20240503 08:12:33.055109 27356 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13767, Committed index: 13767, Last appended: 3.13768, Last appended by leader: 13768, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.055774 26995 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af reported cstate change: term changed from 5 to 6, leader changed from d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195) to 42e3d7884bb84e129ded61a8271632af (127.16.212.193). New cstate: current_term: 6 leader_uuid: "42e3d7884bb84e129ded61a8271632af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.063766 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 6 FOLLOWER]: Rejecting Update request from peer d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 5. Current term is 6. Ops: []
I20240503 08:12:33.064074 27366 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13768, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.064685 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 LEADER]: Rejecting Update request from peer d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 5. Current term is 6. Ops: []
I20240503 08:12:33.064900 27364 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13768, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.064962 27364 consensus_queue.cc:1225] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Peer 42e3d7884bb84e129ded61a8271632af log is divergent from this leader: its last log entry 6.13769 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13767
W20240503 08:12:33.070832 27355 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 6 is higher.
I20240503 08:12:33.082959 27181 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 6 FOLLOWER]: Refusing update from remote peer 42e3d7884bb84e129ded61a8271632af: Log matching property violated. Preceding OpId in replica: term: 3 index: 13767. Preceding OpId from leader: term: 6 index: 13769. (index mismatch)
I20240503 08:12:33.083290 27355 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13769, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.088958 27255 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 08:12:33.089023 27255 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d2620a06bcb6443f8bd1f4f48bc4e0f8, State: Running, Role: LEADER
I20240503 08:12:33.089072 27255 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13767, Committed index: 13767, Last appended: 5.13768, Last appended by leader: 13768, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.089171 27255 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 08:12:33.089812 27255 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 6 FOLLOWER]: Refusing update from remote peer 42e3d7884bb84e129ded61a8271632af: Log matching property violated. Preceding OpId in replica: term: 5 index: 13768. Preceding OpId from leader: term: 6 index: 13769. (index mismatch)
I20240503 08:12:33.090087 27355 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13769, Last known committed idx: 13767, Time since last communication: 0.000s
I20240503 08:12:33.090160 27355 consensus_queue.cc:1225] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer d2620a06bcb6443f8bd1f4f48bc4e0f8 log is divergent from this leader: its last log entry 5.13768 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13767
I20240503 08:12:33.090443 27255 pending_rounds.cc:76] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: Aborting all ops after (but not including) 13767
I20240503 08:12:33.090493 27255 pending_rounds.cc:97] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: Aborting uncommitted NO_OP operation due to leader change: 5.13768
I20240503 08:12:33.090517 27255 raft_consensus.cc:2866] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:33.115011 27328 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42e3d7884bb84e129ded61a8271632af)
I20240503 08:12:33.115109 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:12:33.116114 27328 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.116292 27328 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 7 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.116557 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 7 candidate_status { last_received { term: 6 index: 13769 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.116568 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 7 candidate_status { last_received { term: 6 index: 13769 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.117137 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca; no voters: 42e3d7884bb84e129ded61a8271632af, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.117321 27328 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:12:33.121003 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 6. Current term is 7. Ops: []
I20240503 08:12:33.121246 27355 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: INVALID_TERM, Last received: 6.13769, Next index: 13770, Last known committed idx: 13769, Time since last communication: 0.000s
I20240503 08:12:33.121613 27364 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42e3d7884bb84e129ded61a8271632af)
I20240503 08:12:33.121692 27364 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:12:33.122738 27364 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.122885 27364 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 7 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:33.123046 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 7 candidate_status { last_received { term: 6 index: 13769 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.123133 27181 raft_consensus.cc:2379] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 in current term 7: Already voted for candidate 5a436c985aca40adbd15dbee459a01ca in this term.
I20240503 08:12:33.123178 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 7 candidate_status { last_received { term: 6 index: 13769 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.123334 27221 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca
I20240503 08:12:33.123448 27364 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 08:12:33.124421 27255 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 7 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 6. Current term is 7. Ops: [6.13770-6.13770]
I20240503 08:12:33.124625 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 6. Current term is 7. Ops: [6.13770-6.13770]
I20240503 08:12:33.124641 27365 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 6.13769, Next index: 13770, Last known committed idx: 13769, Time since last communication: 0.000s
I20240503 08:12:33.124843 27355 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: INVALID_TERM, Last received: 6.13769, Next index: 13770, Last known committed idx: 13769, Time since last communication: 0.000s
I20240503 08:12:33.151933 27328 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.152040 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:12:33.152933 27328 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.153283 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 8 candidate_status { last_received { term: 6 index: 13769 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.153304 27328 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 8 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.153373 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 8 candidate_status { last_received { term: 6 index: 13769 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.153458 27255 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:12:33.154064 27255 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a436c985aca40adbd15dbee459a01ca in term 8.
I20240503 08:12:33.154243 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af
I20240503 08:12:33.154352 27328 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 FOLLOWER]: Leader election won for term 8
I20240503 08:12:33.154428 27328 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 LEADER]: Becoming Leader. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Running, Role: LEADER
I20240503 08:12:33.154572 27328 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13769, Committed index: 13769, Last appended: 6.13769, Last appended by leader: 13769, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.155054 26995 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca reported cstate change: term changed from 6 to 8, leader changed from 42e3d7884bb84e129ded61a8271632af (127.16.212.193) to 5a436c985aca40adbd15dbee459a01ca (127.16.212.194). New cstate: current_term: 8 leader_uuid: "5a436c985aca40adbd15dbee459a01ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.156777 27255 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 8 FOLLOWER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 6. Current term is 8. Ops: [6.13770-6.13770]
I20240503 08:12:33.157037 27365 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 6.13769, Next index: 13770, Last known committed idx: 13769, Time since last communication: 0.000s
I20240503 08:12:33.159487 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 LEADER]: Rejecting Update request from peer 42e3d7884bb84e129ded61a8271632af for earlier term 6. Current term is 8. Ops: [6.13770-6.13770]
I20240503 08:12:33.159526 27356 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 LEADER]: Stepping down as leader of term 6
I20240503 08:12:33.159585 27356 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42e3d7884bb84e129ded61a8271632af, State: Running, Role: LEADER
I20240503 08:12:33.159633 27356 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13769, Committed index: 13769, Last appended: 6.13770, Last appended by leader: 13770, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.159726 27356 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 6 FOLLOWER]: Advancing to term 7
I20240503 08:12:33.177521 27255 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 8 FOLLOWER]: Refusing update from remote peer 5a436c985aca40adbd15dbee459a01ca: Log matching property violated. Preceding OpId in replica: term: 6 index: 13769. Preceding OpId from leader: term: 8 index: 13770. (index mismatch)
I20240503 08:12:33.177870 27344 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13770, Last known committed idx: 13769, Time since last communication: 0.000s
I20240503 08:12:33.177984 27105 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 7 FOLLOWER]: Advancing to term 8
I20240503 08:12:33.178992 27105 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 8 FOLLOWER]: Refusing update from remote peer 5a436c985aca40adbd15dbee459a01ca: Log matching property violated. Preceding OpId in replica: term: 6 index: 13770. Preceding OpId from leader: term: 8 index: 13770. (term mismatch)
I20240503 08:12:33.179036 27105 pending_rounds.cc:76] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: Aborting all ops after (but not including) 13769
I20240503 08:12:33.179050 27105 pending_rounds.cc:97] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 6.13770
W20240503 08:12:33.179080 27105 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33906: Aborted: Op aborted by new leader
I20240503 08:12:33.179297 27328 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13770, Last known committed idx: 13769, Time since last communication: 0.000s
I20240503 08:12:33.264883 27355 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5a436c985aca40adbd15dbee459a01ca)
I20240503 08:12:33.264986 27355 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 8 FOLLOWER]: Advancing to term 9
I20240503 08:12:33.265684 27355 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.265869 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 9 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 8. Current term is 9. Ops: []
I20240503 08:12:33.265872 27355 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 9 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.266010 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 9 candidate_status { last_received { term: 8 index: 13771 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.266062 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 9 candidate_status { last_received { term: 8 index: 13771 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.266115 27328 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 8.13771, Next index: 13772, Last known committed idx: 13771, Time since last communication: 0.000s
I20240503 08:12:33.266276 27071 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af; no voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.266386 27355 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240503 08:12:33.275388 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 9 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 8. Current term is 9. Ops: [8.13772-8.13772]
I20240503 08:12:33.275627 27312 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 8.13771, Next index: 13772, Last known committed idx: 13771, Time since last communication: 0.000s
I20240503 08:12:33.305014 27356 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.305122 27356 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 9 FOLLOWER]: Advancing to term 10
I20240503 08:12:33.305794 27356 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.305946 27356 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 10 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.306149 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 10 candidate_status { last_received { term: 8 index: 13771 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.306170 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 10 candidate_status { last_received { term: 8 index: 13771 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.306479 27070 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af; no voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.306573 27356 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240503 08:12:33.309685 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 10 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 8. Current term is 10. Ops: [8.13772-8.13772]
I20240503 08:12:33.310009 27344 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 8.13771, Next index: 13772, Last known committed idx: 13771, Time since last communication: 0.000s
I20240503 08:12:33.310690 27328 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 LEADER]: Stepping down as leader of term 8
I20240503 08:12:33.310745 27328 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Running, Role: LEADER
I20240503 08:12:33.310791 27328 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13772, Committed index: 13772, Last appended: 8.13772, Last appended by leader: 13772, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.310881 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 8 FOLLOWER]: Advancing to term 9
I20240503 08:12:33.333179 27348 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.333294 27348 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 9 FOLLOWER]: Advancing to term 10
I20240503 08:12:33.334025 27348 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.334174 27348 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 10 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.334504 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 10 candidate_status { last_received { term: 8 index: 13772 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.334548 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 10 candidate_status { last_received { term: 8 index: 13772 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.334600 27105 raft_consensus.cc:2379] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a436c985aca40adbd15dbee459a01ca in current term 10: Already voted for candidate 42e3d7884bb84e129ded61a8271632af in this term.
I20240503 08:12:33.334642 27255 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 8 FOLLOWER]: Advancing to term 10
I20240503 08:12:33.335265 27255 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a436c985aca40adbd15dbee459a01ca in term 10.
I20240503 08:12:33.335510 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af
I20240503 08:12:33.335659 27348 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 10 FOLLOWER]: Leader election won for term 10
I20240503 08:12:33.335731 27348 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 10 LEADER]: Becoming Leader. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Running, Role: LEADER
I20240503 08:12:33.335794 27348 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13771, Committed index: 13771, Last appended: 8.13772, Last appended by leader: 13772, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.336341 27001 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "5a436c985aca40adbd15dbee459a01ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.348554 27356 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.348662 27356 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 10 FOLLOWER]: Advancing to term 11
I20240503 08:12:33.349310 27356 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.349473 27356 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 11 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.349725 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 11 candidate_status { last_received { term: 8 index: 13771 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.349726 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 11 candidate_status { last_received { term: 8 index: 13771 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.349841 27255 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 10 FOLLOWER]: Advancing to term 11
I20240503 08:12:33.350447 27255 raft_consensus.cc:2396] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e3d7884bb84e129ded61a8271632af for term 11 because replica has last-logged OpId of term: 8 index: 13772, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 13771.
I20240503 08:12:33.350679 27070 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af; no voters: 5a436c985aca40adbd15dbee459a01ca, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.350800 27356 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240503 08:12:33.359791 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 11 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 10. Current term is 11. Ops: []
I20240503 08:12:33.360101 27348 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13773, Last known committed idx: 13771, Time since last communication: 0.000s
W20240503 08:12:33.362684 27328 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 10 is higher.
I20240503 08:12:33.363611 27255 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 11 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 10. Current term is 11. Ops: []
I20240503 08:12:33.363936 27348 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13773, Last known committed idx: 13772, Time since last communication: 0.000s
I20240503 08:12:33.369238 27366 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.369324 27366 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 11 FOLLOWER]: Advancing to term 12
I20240503 08:12:33.369940 27366 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.370077 27366 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 12 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
I20240503 08:12:33.370332 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 12 candidate_status { last_received { term: 8 index: 13772 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.370329 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" candidate_term: 12 candidate_status { last_received { term: 8 index: 13772 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.370427 27105 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 11 FOLLOWER]: Advancing to term 12
I20240503 08:12:33.371040 27105 raft_consensus.cc:2454] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2620a06bcb6443f8bd1f4f48bc4e0f8 in term 12.
I20240503 08:12:33.371228 27221 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42e3d7884bb84e129ded61a8271632af, d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 5a436c985aca40adbd15dbee459a01ca
I20240503 08:12:33.371322 27366 raft_consensus.cc:2781] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 FOLLOWER]: Leader election won for term 12
I20240503 08:12:33.371385 27366 raft_consensus.cc:686] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 LEADER]: Becoming Leader. State: Replica: d2620a06bcb6443f8bd1f4f48bc4e0f8, State: Running, Role: LEADER
I20240503 08:12:33.371448 27366 consensus_queue.cc:237] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13772, Committed index: 13772, Last appended: 8.13772, Last appended by leader: 13772, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.371951 27001 catalog_manager.cc:5484] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 reported cstate change: term changed from 10 to 12, leader changed from 5a436c985aca40adbd15dbee459a01ca (127.16.212.194) to d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195). New cstate: current_term: 12 leader_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:33.374245 27255 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 LEADER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 10. Current term is 12. Ops: [10.13773-10.13773]
I20240503 08:12:33.374241 27105 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 12 FOLLOWER]: Rejecting Update request from peer 5a436c985aca40adbd15dbee459a01ca for earlier term 10. Current term is 12. Ops: [8.13772-10.13773]
I20240503 08:12:33.374657 27348 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: INVALID_TERM, Last received: 8.13771, Next index: 13772, Last known committed idx: 13771, Time since last communication: 0.000s
I20240503 08:12:33.374804 27344 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 }, Status: INVALID_TERM, Last received: 8.13772, Next index: 13773, Last known committed idx: 13772, Time since last communication: 0.000s
I20240503 08:12:33.374866 27344 consensus_queue.cc:1225] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [LEADER]: Peer d2620a06bcb6443f8bd1f4f48bc4e0f8 log is divergent from this leader: its last log entry 12.13773 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13772
W20240503 08:12:33.379021 27328 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20240503 08:12:33.383172 27328 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 10 LEADER]: Stepping down as leader of term 10
I20240503 08:12:33.383215 27328 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5a436c985aca40adbd15dbee459a01ca, State: Running, Role: LEADER
I20240503 08:12:33.383266 27328 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13771, Committed index: 13771, Last appended: 10.13774, Last appended by leader: 13774, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.383397 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 10 FOLLOWER]: Advancing to term 11
I20240503 08:12:33.398046 27181 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 11 FOLLOWER]: Advancing to term 12
I20240503 08:12:33.398350 27105 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 12 FOLLOWER]: Refusing update from remote peer d2620a06bcb6443f8bd1f4f48bc4e0f8: Log matching property violated. Preceding OpId in replica: term: 8 index: 13771. Preceding OpId from leader: term: 12 index: 13773. (index mismatch)
I20240503 08:12:33.398589 27366 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13773, Last known committed idx: 13771, Time since last communication: 0.000s
I20240503 08:12:33.399034 27181 raft_consensus.cc:1261] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 12 FOLLOWER]: Refusing update from remote peer d2620a06bcb6443f8bd1f4f48bc4e0f8: Log matching property violated. Preceding OpId in replica: term: 10 index: 13774. Preceding OpId from leader: term: 12 index: 13773. (term mismatch)
I20240503 08:12:33.399076 27181 pending_rounds.cc:76] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Aborting all ops after (but not including) 13772
I20240503 08:12:33.399096 27181 pending_rounds.cc:97] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Aborting uncommitted NO_OP operation due to leader change: 10.13773
I20240503 08:12:33.399114 27181 raft_consensus.cc:2866] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 08:12:33.399137 27181 pending_rounds.cc:97] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 10.13774
W20240503 08:12:33.399160 27181 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60650: Aborted: Op aborted by new leader
I20240503 08:12:33.399360 27364 consensus_queue.cc:1035] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13773, Last known committed idx: 13771, Time since last communication: 0.000s
W20240503 08:12:33.404870 27095 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33906: Illegal state: replica 42e3d7884bb84e129ded61a8271632af is not leader of this config: current role FOLLOWER
I20240503 08:12:33.444415 27348 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 12 FOLLOWER]: Starting leader election (detected failure of leader d2620a06bcb6443f8bd1f4f48bc4e0f8)
I20240503 08:12:33.444519 27348 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 12 FOLLOWER]: Advancing to term 13
I20240503 08:12:33.445238 27348 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.445386 27348 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 13 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.445631 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 13 candidate_status { last_received { term: 12 index: 13774 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.445645 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 13 candidate_status { last_received { term: 12 index: 13774 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.445875 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca; no voters: 42e3d7884bb84e129ded61a8271632af, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.445957 27348 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20240503 08:12:33.446130 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 13 FOLLOWER]: Rejecting Update request from peer d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 12. Current term is 13. Ops: []
I20240503 08:12:33.446643 27367 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: INVALID_TERM, Last received: 12.13774, Next index: 13775, Last known committed idx: 13774, Time since last communication: 0.000s
I20240503 08:12:33.474346 27181 raft_consensus.cc:1226] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 13 FOLLOWER]: Rejecting Update request from peer d2620a06bcb6443f8bd1f4f48bc4e0f8 for earlier term 12. Current term is 13. Ops: []
I20240503 08:12:33.474694 27364 consensus_queue.cc:1046] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 }, Status: INVALID_TERM, Last received: 12.13774, Next index: 13775, Last known committed idx: 13774, Time since last communication: 0.000s
I20240503 08:12:33.485883 27328 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.485987 27328 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 13 FOLLOWER]: Advancing to term 14
I20240503 08:12:33.486712 27328 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.486876 27328 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 14 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.487162 27105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 14 candidate_status { last_received { term: 12 index: 13774 } } ignore_live_leader: false dest_uuid: "42e3d7884bb84e129ded61a8271632af"
I20240503 08:12:33.487185 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "5a436c985aca40adbd15dbee459a01ca" candidate_term: 14 candidate_status { last_received { term: 12 index: 13774 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.487540 27146 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a436c985aca40adbd15dbee459a01ca; no voters: 42e3d7884bb84e129ded61a8271632af, d2620a06bcb6443f8bd1f4f48bc4e0f8
I20240503 08:12:33.487651 27328 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240503 08:12:33.497623 27356 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 12 FOLLOWER]: Starting leader election (detected failure of leader d2620a06bcb6443f8bd1f4f48bc4e0f8)
I20240503 08:12:33.497722 27356 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 12 FOLLOWER]: Advancing to term 13
I20240503 08:12:33.498508 27356 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.498699 27356 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 13 election: Requested vote from peers 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047), d2620a06bcb6443f8bd1f4f48bc4e0f8 (127.16.212.195:45571)
I20240503 08:12:33.498871 27181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 13 candidate_status { last_received { term: 12 index: 13774 } } ignore_live_leader: false dest_uuid: "5a436c985aca40adbd15dbee459a01ca"
I20240503 08:12:33.498930 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:12:33.498965 27181 raft_consensus.cc:2354] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e3d7884bb84e129ded61a8271632af for earlier term 13. Current term is 14.
I20240503 08:12:33.499552 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfda1db397144737b1311452c50846fd" candidate_uuid: "42e3d7884bb84e129ded61a8271632af" candidate_term: 13 candidate_status { last_received { term: 12 index: 13774 } } ignore_live_leader: false dest_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8"
I20240503 08:12:33.499646 27071 leader_election.cc:398] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 13 election: Vote denied by peer 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047) with higher term. Message: Invalid argument: T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e3d7884bb84e129ded61a8271632af for earlier term 13. Current term is 14.
I20240503 08:12:33.500087 27071 leader_election.cc:401] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20240503 08:12:33.500185 27356 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 13 FOLLOWER]: Advancing to term 14
I20240503 08:12:33.500229 27367 raft_consensus.cc:3032] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 LEADER]: Stepping down as leader of term 12
I20240503 08:12:33.500284 27367 raft_consensus.cc:729] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: d2620a06bcb6443f8bd1f4f48bc4e0f8, State: Running, Role: LEADER
I20240503 08:12:33.500349 27367 consensus_queue.cc:260] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13774, Committed index: 13774, Last appended: 12.13774, Last appended by leader: 13774, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.500464 27367 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 12 FOLLOWER]: Advancing to term 13
I20240503 08:12:33.501271 27356 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047) with higher term. Message: Invalid argument: T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e3d7884bb84e129ded61a8271632af for earlier term 13. Current term is 14.
I20240503 08:12:33.502790 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:33.502914 17235 tablet_replica.cc:330] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af: stopping tablet replica
I20240503 08:12:33.502979 17235 raft_consensus.cc:2229] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 14 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.503028 17235 raft_consensus.cc:2258] T dfda1db397144737b1311452c50846fd P 42e3d7884bb84e129ded61a8271632af [term 14 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.508152 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:12:33.509093 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:12:33.513008 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:33.513202 17235 tablet_replica.cc:330] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca: stopping tablet replica
I20240503 08:12:33.513268 17235 raft_consensus.cc:2229] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.513329 17235 raft_consensus.cc:2258] T dfda1db397144737b1311452c50846fd P 5a436c985aca40adbd15dbee459a01ca [term 14 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.532893 27364 raft_consensus.cc:482] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.533017 27364 raft_consensus.cc:3037] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 13 FOLLOWER]: Advancing to term 14
I20240503 08:12:33.533501 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:12:33.533964 27364 raft_consensus.cc:504] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3d7884bb84e129ded61a8271632af" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 41641 } } peers { permanent_uuid: "5a436c985aca40adbd15dbee459a01ca" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 45047 } } peers { permanent_uuid: "d2620a06bcb6443f8bd1f4f48bc4e0f8" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45571 } }
I20240503 08:12:33.534147 27364 leader_election.cc:288] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 14 election: Requested vote from peers 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641), 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047)
W20240503 08:12:33.534498 27221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.212.193:41641: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 08:12:33.534720 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
W20240503 08:12:33.534893 27221 leader_election.cc:334] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 42e3d7884bb84e129ded61a8271632af (127.16.212.193:41641): Network error: Client connection negotiation failed: client connection to 127.16.212.193:41641: connect: Connection refused (error 111)
W20240503 08:12:33.534999 27222 leader_election.cc:334] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 5a436c985aca40adbd15dbee459a01ca (127.16.212.194:45047): Network error: Client connection negotiation failed: client connection to 127.16.212.194:45047: connect: Connection refused (error 111)
I20240503 08:12:33.535069 27222 leader_election.cc:302] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2620a06bcb6443f8bd1f4f48bc4e0f8; no voters: 42e3d7884bb84e129ded61a8271632af, 5a436c985aca40adbd15dbee459a01ca
I20240503 08:12:33.535156 27364 raft_consensus.cc:2726] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20240503 08:12:33.538964 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:33.539072 17235 tablet_replica.cc:330] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8: stopping tablet replica
I20240503 08:12:33.539144 17235 raft_consensus.cc:2229] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 14 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:33.558858 17235 raft_consensus.cc:2258] T dfda1db397144737b1311452c50846fd P d2620a06bcb6443f8bd1f4f48bc4e0f8 [term 14 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.584440 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:12:33.585614 17235 master.cc:537] Master@127.16.212.254:39025 shutting down...
I20240503 08:12:33.588241 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:33.588328 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:33.588361 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6281f4184fd942d09017a4d9c30bf205: stopping tablet replica
I20240503 08:12:33.600201 17235 master.cc:552] Master@127.16.212.254:39025 shutdown complete.
[ OK ] TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm (10956 ms)
[----------] 2 tests from TxnParticipantElectionStormITest (41289 ms total)
[----------] 2 tests from ShouldGCWals/ParticipantCopyITest
[ RUN ] ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps/0
I20240503 08:12:33.621618 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:40439
I20240503 08:12:33.621857 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:33.624354 27372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:33.625037 27373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.625838 17235 server_base.cc:1034] running on GCE node
W20240503 08:12:33.626443 27375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.626600 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:33.626679 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:33.626708 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723953626707 us; error 0 us; skew 500 ppm
I20240503 08:12:33.627470 17235 webserver.cc:460] Webserver started at http://127.16.212.254:38751/ using document root <none> and password file <none>
I20240503 08:12:33.627562 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:33.627597 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:33.627627 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:33.628144 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "e4539b3eedcc4ea4aa3566df9db5aa81"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.629904 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:12:33.630570 27380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.630748 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:33.630815 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "e4539b3eedcc4ea4aa3566df9db5aa81"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.630867 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:33.659519 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:33.659760 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:33.662766 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:40439
I20240503 08:12:33.665438 27432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:40439 every 8 connection(s)
I20240503 08:12:33.667538 27433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.668591 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81: Bootstrap starting.
I20240503 08:12:33.668838 27433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.669399 27433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81: No bootstrap required, opened a new log
I20240503 08:12:33.669499 27433 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" member_type: VOTER }
I20240503 08:12:33.669549 27433 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.669562 27433 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4539b3eedcc4ea4aa3566df9db5aa81, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.669602 27433 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" member_type: VOTER }
I20240503 08:12:33.669641 27433 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:33.669662 27433 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:33.669692 27433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.670153 27433 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" member_type: VOTER }
I20240503 08:12:33.670210 27433 leader_election.cc:302] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4539b3eedcc4ea4aa3566df9db5aa81; no voters:
I20240503 08:12:33.670334 27433 leader_election.cc:288] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:33.670396 27436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:33.670595 27436 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 1 LEADER]: Becoming Leader. State: Replica: e4539b3eedcc4ea4aa3566df9db5aa81, State: Running, Role: LEADER
I20240503 08:12:33.670678 27436 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" member_type: VOTER }
I20240503 08:12:33.670681 27433 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:33.670893 27437 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" member_type: VOTER } }
I20240503 08:12:33.670902 27438 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4539b3eedcc4ea4aa3566df9db5aa81. Latest consensus state: current_term: 1 leader_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4539b3eedcc4ea4aa3566df9db5aa81" member_type: VOTER } }
I20240503 08:12:33.670969 27437 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:33.670981 27438 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:33.671320 27441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:33.671519 27441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:33.672122 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:33.672236 27441 catalog_manager.cc:1347] Generated new cluster ID: b862736dc66a4821a0caa79060883173
I20240503 08:12:33.672279 27441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:33.683413 27441 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:33.683881 27441 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:33.688092 27441 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81: Generated new TSK 0
I20240503 08:12:33.688200 27441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:33.704206 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:33.706712 27454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:33.707357 27455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:33.708285 27457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.708854 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:33.708976 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:33.709014 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:33.709041 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723953709041 us; error 0 us; skew 500 ppm
I20240503 08:12:33.709766 17235 webserver.cc:460] Webserver started at http://127.16.212.193:36043/ using document root <none> and password file <none>
I20240503 08:12:33.709865 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:33.709924 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:33.709977 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:33.710289 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "9bd13ada0adf49b994fa8c4a425744f1"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.711388 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:33.711907 27462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.712244 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:33.712345 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "9bd13ada0adf49b994fa8c4a425744f1"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.712419 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:33.719895 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:33.720163 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:33.720335 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:33.720533 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:33.720561 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.720594 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:33.720623 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.723681 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:40597
I20240503 08:12:33.723722 27525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:40597 every 8 connection(s)
I20240503 08:12:33.724087 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:33.727912 27530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.727919 27526 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40439
I20240503 08:12:33.728016 27526 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:33.728127 27526 heartbeater.cc:511] Master 127.16.212.254:40439 requested a full tablet report, sending...
I20240503 08:12:33.728402 27397 ts_manager.cc:194] Registered new tserver with Master: 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193:40597)
W20240503 08:12:33.728801 27531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.729125 27397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45294
W20240503 08:12:33.729384 27533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.730098 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:33.730357 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:33.730403 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:33.730437 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723953730437 us; error 0 us; skew 500 ppm
I20240503 08:12:33.732062 17235 webserver.cc:460] Webserver started at http://127.16.212.194:42767/ using document root <none> and password file <none>
I20240503 08:12:33.732173 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:33.732262 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:33.732316 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:33.732604 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "f83e30770df64414b2a76bb20ff27be6"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.733660 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:33.734421 27538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.734853 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 08:12:33.734913 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "f83e30770df64414b2a76bb20ff27be6"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.734951 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:33.772130 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:33.772348 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:33.772511 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:33.772708 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:33.772735 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.772766 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:33.772791 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.775552 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:42371
I20240503 08:12:33.775671 27601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:42371 every 8 connection(s)
I20240503 08:12:33.775918 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:33.778270 27606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:33.779018 27608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.779079 27602 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40439
I20240503 08:12:33.779147 27602 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:33.779263 27602 heartbeater.cc:511] Master 127.16.212.254:40439 requested a full tablet report, sending...
I20240503 08:12:33.779520 27397 ts_manager.cc:194] Registered new tserver with Master: f83e30770df64414b2a76bb20ff27be6 (127.16.212.194:42371)
W20240503 08:12:33.779541 27605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:33.779914 27397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45306
I20240503 08:12:33.780215 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:33.780349 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:33.780385 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:33.780405 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723953780405 us; error 0 us; skew 500 ppm
I20240503 08:12:33.781069 17235 webserver.cc:460] Webserver started at http://127.16.212.195:44501/ using document root <none> and password file <none>
I20240503 08:12:33.781138 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:33.781176 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:33.781204 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:33.781452 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "810dfe154e0d4edfa5625eea399613c4"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.782310 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:33.782764 27613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.782860 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.782907 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "810dfe154e0d4edfa5625eea399613c4"
format_stamp: "Formatted at 2024-05-03 08:12:33 on dist-test-slave-zl2t"
I20240503 08:12:33.782948 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:33.788537 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:33.788733 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:33.788885 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:33.789081 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:33.789108 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.789137 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:33.789158 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.792471 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:37211
I20240503 08:12:33.792513 27676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:37211 every 8 connection(s)
I20240503 08:12:33.795552 27677 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40439
I20240503 08:12:33.795610 27677 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:33.795719 27677 heartbeater.cc:511] Master 127.16.212.254:40439 requested a full tablet report, sending...
I20240503 08:12:33.795966 27397 ts_manager.cc:194] Registered new tserver with Master: 810dfe154e0d4edfa5625eea399613c4 (127.16.212.195:37211)
I20240503 08:12:33.795974 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003362959s
I20240503 08:12:33.796396 17235 test_util.cc:268] Using random seed: 1210541356
I20240503 08:12:33.796463 27397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45310
I20240503 08:12:33.801810 27397 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45312:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:12:33.801967 27397 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:12:33.807332 27566 tablet_service.cc:1467] Processing CreateTablet for tablet 1c30cc6bf11a44b987f50663b2f102c0 (DEFAULT_TABLE table=test-workload [id=bee9b1077a68409bbd6ac29c5720812c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.807564 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30cc6bf11a44b987f50663b2f102c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.807369 27490 tablet_service.cc:1467] Processing CreateTablet for tablet 1c30cc6bf11a44b987f50663b2f102c0 (DEFAULT_TABLE table=test-workload [id=bee9b1077a68409bbd6ac29c5720812c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.807673 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 1c30cc6bf11a44b987f50663b2f102c0 (DEFAULT_TABLE table=test-workload [id=bee9b1077a68409bbd6ac29c5720812c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:33.807761 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30cc6bf11a44b987f50663b2f102c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.807780 27490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30cc6bf11a44b987f50663b2f102c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:33.809226 27693 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Bootstrap starting.
I20240503 08:12:33.809342 27692 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Bootstrap starting.
I20240503 08:12:33.809751 27692 tablet_bootstrap.cc:654] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.809751 27693 tablet_bootstrap.cc:654] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.809810 27694 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Bootstrap starting.
I20240503 08:12:33.810262 27694 tablet_bootstrap.cc:654] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:33.810410 27692 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: No bootstrap required, opened a new log
I20240503 08:12:33.810443 27693 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: No bootstrap required, opened a new log
I20240503 08:12:33.810467 27692 ts_tablet_manager.cc:1367] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:33.810488 27693 ts_tablet_manager.cc:1367] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:33.810667 27692 raft_consensus.cc:348] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.810685 27693 raft_consensus.cc:348] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.810770 27692 raft_consensus.cc:374] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.810786 27693 raft_consensus.cc:374] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.810791 27692 raft_consensus.cc:729] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd13ada0adf49b994fa8c4a425744f1, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.810835 27693 raft_consensus.cc:729] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810dfe154e0d4edfa5625eea399613c4, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.810863 27692 consensus_queue.cc:260] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.810921 27693 consensus_queue.cc:260] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.811028 27692 ts_tablet_manager.cc:1398] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:33.811060 27526 heartbeater.cc:503] Master 127.16.212.254:40439 was elected leader, sending a full tablet report...
I20240503 08:12:33.811094 27693 ts_tablet_manager.cc:1398] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:33.811156 27677 heartbeater.cc:503] Master 127.16.212.254:40439 was elected leader, sending a full tablet report...
I20240503 08:12:33.811173 27694 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: No bootstrap required, opened a new log
I20240503 08:12:33.811295 27694 ts_tablet_manager.cc:1367] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 08:12:33.811466 27694 raft_consensus.cc:348] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.811532 27694 raft_consensus.cc:374] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:33.811554 27694 raft_consensus.cc:729] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83e30770df64414b2a76bb20ff27be6, State: Initialized, Role: FOLLOWER
I20240503 08:12:33.811594 27694 consensus_queue.cc:260] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.811756 27694 ts_tablet_manager.cc:1398] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:33.811906 27602 heartbeater.cc:503] Master 127.16.212.254:40439 was elected leader, sending a full tablet report...
W20240503 08:12:33.812950 27678 tablet.cc:2369] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 08:12:33.964627 27700 raft_consensus.cc:482] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:33.964749 27700 raft_consensus.cc:504] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.964980 27700 leader_election.cc:288] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193:40597), 810dfe154e0d4edfa5625eea399613c4 (127.16.212.195:37211)
I20240503 08:12:33.967825 27699 raft_consensus.cc:482] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:33.967911 27699 raft_consensus.cc:504] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.968091 27699 leader_election.cc:288] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193:40597), f83e30770df64414b2a76bb20ff27be6 (127.16.212.194:42371)
I20240503 08:12:33.968601 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "f83e30770df64414b2a76bb20ff27be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810dfe154e0d4edfa5625eea399613c4" is_pre_election: true
I20240503 08:12:33.968681 27651 raft_consensus.cc:2454] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83e30770df64414b2a76bb20ff27be6 in term 0.
I20240503 08:12:33.968845 27541 leader_election.cc:302] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810dfe154e0d4edfa5625eea399613c4, f83e30770df64414b2a76bb20ff27be6; no voters:
I20240503 08:12:33.968925 27700 raft_consensus.cc:2781] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:33.968693 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "f83e30770df64414b2a76bb20ff27be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd13ada0adf49b994fa8c4a425744f1" is_pre_election: true
I20240503 08:12:33.968966 27700 raft_consensus.cc:482] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.969089 27500 raft_consensus.cc:2454] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83e30770df64414b2a76bb20ff27be6 in term 0.
I20240503 08:12:33.969091 27700 raft_consensus.cc:3037] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.971238 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "810dfe154e0d4edfa5625eea399613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd13ada0adf49b994fa8c4a425744f1" is_pre_election: true
I20240503 08:12:33.971256 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "810dfe154e0d4edfa5625eea399613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83e30770df64414b2a76bb20ff27be6" is_pre_election: true
I20240503 08:12:33.971333 27500 raft_consensus.cc:2454] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 810dfe154e0d4edfa5625eea399613c4 in term 0.
I20240503 08:12:33.971477 27615 leader_election.cc:302] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 810dfe154e0d4edfa5625eea399613c4, 9bd13ada0adf49b994fa8c4a425744f1; no voters:
I20240503 08:12:33.971594 27699 raft_consensus.cc:2781] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:33.971637 27699 raft_consensus.cc:482] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:33.971681 27699 raft_consensus.cc:3037] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.971644 27700 raft_consensus.cc:504] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.971781 27576 raft_consensus.cc:2379] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 810dfe154e0d4edfa5625eea399613c4 in current term 1: Already voted for candidate f83e30770df64414b2a76bb20ff27be6 in this term.
I20240503 08:12:33.971928 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "f83e30770df64414b2a76bb20ff27be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "810dfe154e0d4edfa5625eea399613c4"
I20240503 08:12:33.971880 27700 leader_election.cc:288] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [CANDIDATE]: Term 1 election: Requested vote from peers 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193:40597), 810dfe154e0d4edfa5625eea399613c4 (127.16.212.195:37211)
I20240503 08:12:33.971975 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "f83e30770df64414b2a76bb20ff27be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd13ada0adf49b994fa8c4a425744f1"
I20240503 08:12:33.972154 27500 raft_consensus.cc:3037] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:33.972478 27699 raft_consensus.cc:504] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.972601 27699 leader_election.cc:288] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [CANDIDATE]: Term 1 election: Requested vote from peers 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193:40597), f83e30770df64414b2a76bb20ff27be6 (127.16.212.194:42371)
I20240503 08:12:33.972601 27651 raft_consensus.cc:2379] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f83e30770df64414b2a76bb20ff27be6 in current term 1: Already voted for candidate 810dfe154e0d4edfa5625eea399613c4 in this term.
I20240503 08:12:33.972803 27499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "810dfe154e0d4edfa5625eea399613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd13ada0adf49b994fa8c4a425744f1"
I20240503 08:12:33.972865 27500 raft_consensus.cc:2454] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83e30770df64414b2a76bb20ff27be6 in term 1.
I20240503 08:12:33.972900 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "810dfe154e0d4edfa5625eea399613c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83e30770df64414b2a76bb20ff27be6"
I20240503 08:12:33.972950 27576 raft_consensus.cc:2379] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 810dfe154e0d4edfa5625eea399613c4 in current term 1: Already voted for candidate f83e30770df64414b2a76bb20ff27be6 in this term.
I20240503 08:12:33.973100 27614 leader_election.cc:302] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 810dfe154e0d4edfa5625eea399613c4; no voters: 9bd13ada0adf49b994fa8c4a425744f1, f83e30770df64414b2a76bb20ff27be6
I20240503 08:12:33.973117 27540 leader_election.cc:302] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bd13ada0adf49b994fa8c4a425744f1, f83e30770df64414b2a76bb20ff27be6; no voters: 810dfe154e0d4edfa5625eea399613c4
I20240503 08:12:33.973194 27700 raft_consensus.cc:2781] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:33.973228 27699 raft_consensus.cc:2726] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 08:12:33.973340 27700 raft_consensus.cc:686] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 LEADER]: Becoming Leader. State: Replica: f83e30770df64414b2a76bb20ff27be6, State: Running, Role: LEADER
I20240503 08:12:33.973407 27700 consensus_queue.cc:237] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:33.974006 27397 catalog_manager.cc:5484] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f83e30770df64414b2a76bb20ff27be6 (127.16.212.194). New cstate: current_term: 1 leader_uuid: "f83e30770df64414b2a76bb20ff27be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:33.997787 27651 raft_consensus.cc:1261] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 1 FOLLOWER]: Refusing update from remote peer f83e30770df64414b2a76bb20ff27be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:33.997825 27500 raft_consensus.cc:1261] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 1 FOLLOWER]: Refusing update from remote peer f83e30770df64414b2a76bb20ff27be6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:33.998123 27704 consensus_queue.cc:1035] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:33.998225 27700 consensus_queue.cc:1035] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:34.141047 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.147670 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.150154 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.164615 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":280346,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1032,"drs_written":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:34.164820 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 90024 bytes on disk
I20240503 08:12:34.166436 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.168516 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.027s user 0.015s sys 0.004s Metrics: {"bytes_written":273775,"cfile_init":1,"compiler_manager_pool.queue_time_us":14377,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":867,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500,"spinlock_wait_cycles":3328,"thread_start_us":96,"threads_started":1}
I20240503 08:12:34.168834 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 87614 bytes on disk
I20240503 08:12:34.169070 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":29,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.169822 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.012s sys 0.003s Metrics: {"bytes_written":279167,"cfile_init":1,"dirs.queue_time_us":681,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":700,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:12:34.174729 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 89513 bytes on disk
I20240503 08:12:34.174945 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":29,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.342943 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.358069 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.359122 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.367750 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.015s sys 0.000s Metrics: {"bytes_written":271100,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1081,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1478,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20240503 08:12:34.368016 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014771
I20240503 08:12:34.378314 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":281577,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":941,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:34.378587 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014765
I20240503 08:12:34.379902 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":283047,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":872,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":672,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20240503 08:12:34.380028 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 90789 bytes on disk
I20240503 08:12:34.380542 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.380642 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014764
I20240503 08:12:34.440832 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.059s user 0.017s sys 0.011s Metrics: {"bytes_written":559273,"cfile_cache_hit":13,"cfile_cache_hit_bytes":461205,"cfile_cache_miss":39,"cfile_cache_miss_bytes":926781,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":902,"drs_written":1,"lbm_read_time_us":2596,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":58,"lbm_write_time_us":1246,"lbm_writes_lt_1ms":44,"num_input_rowsets":2,"peak_mem_usage":293779,"rows_written":19600,"spinlock_wait_cycles":164480,"thread_start_us":237,"threads_started":5}
I20240503 08:12:34.441033 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 186596 bytes on disk
I20240503 08:12:34.442430 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":59648}
I20240503 08:12:34.443998 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.076s user 0.026s sys 0.000s Metrics: {"bytes_written":541072,"cfile_cache_hit":12,"cfile_cache_hit_bytes":415529,"cfile_cache_miss":40,"cfile_cache_miss_bytes":922157,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":874,"drs_written":1,"lbm_read_time_us":599,"lbm_reads_lt_1ms":64,"lbm_write_time_us":1142,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":283109,"rows_written":18900,"thread_start_us":286,"threads_started":5}
I20240503 08:12:34.444696 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 179583 bytes on disk
I20240503 08:12:34.450887 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.452260 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.073s user 0.028s sys 0.000s Metrics: {"bytes_written":556797,"cfile_cache_hit":13,"cfile_cache_hit_bytes":458832,"cfile_cache_miss":39,"cfile_cache_miss_bytes":921940,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":1108,"drs_written":1,"lbm_read_time_us":1507,"lbm_reads_lt_1ms":63,"lbm_write_time_us":2544,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":292259,"rows_written":19500,"spinlock_wait_cycles":384}
I20240503 08:12:34.452486 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 185345 bytes on disk
I20240503 08:12:34.452775 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.532217 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.544080 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.546043 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":279184,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":914,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":619,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:12:34.546281 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014570
I20240503 08:12:34.554018 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.567719 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.024s user 0.010s sys 0.000s Metrics: {"bytes_written":273742,"cfile_init":1,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":4894,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:12:34.567883 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014564
I20240503 08:12:34.573552 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.019s user 0.011s sys 0.001s Metrics: {"bytes_written":276423,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":912,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:12:34.573724 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014562
I20240503 08:12:34.588563 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.042s user 0.028s sys 0.004s Metrics: {"bytes_written":813125,"cfile_cache_hit":10,"cfile_cache_hit_bytes":445287,"cfile_cache_miss":52,"cfile_cache_miss_bytes":1584053,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":571,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":799,"drs_written":1,"lbm_read_time_us":889,"lbm_reads_lt_1ms":76,"lbm_write_time_us":1649,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":431016,"rows_written":28600}
I20240503 08:12:34.588755 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 281427 bytes on disk
I20240503 08:12:34.588893 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.612099 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.044s user 0.036s sys 0.001s Metrics: {"bytes_written":823481,"cfile_cache_hit":12,"cfile_cache_hit_bytes":558478,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1498684,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":1052,"drs_written":1,"lbm_read_time_us":976,"lbm_reads_lt_1ms":74,"lbm_write_time_us":1717,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":437116,"rows_written":29000}
I20240503 08:12:34.612380 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 285557 bytes on disk
I20240503 08:12:34.612512 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.622028 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.048s user 0.035s sys 0.003s Metrics: {"bytes_written":832449,"cfile_cache_hit":12,"cfile_cache_hit_bytes":560733,"cfile_cache_miss":52,"cfile_cache_miss_bytes":1511077,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":269,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":1170,"drs_written":1,"lbm_read_time_us":1975,"lbm_reads_lt_1ms":76,"lbm_write_time_us":1770,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":440159,"rows_written":29200}
I20240503 08:12:34.622210 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 287696 bytes on disk
I20240503 08:12:34.622310 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.678335 27386 maintenance_manager.cc:419] P e4539b3eedcc4ea4aa3566df9db5aa81: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.003577
I20240503 08:12:34.681993 27385 maintenance_manager.cc:643] P e4539b3eedcc4ea4aa3566df9db5aa81: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.000s sys 0.003s Metrics: {"bytes_written":6637,"cfile_init":1,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1106,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":182,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":5,"thread_start_us":119,"threads_started":1}
I20240503 08:12:34.682160 27386 maintenance_manager.cc:419] P e4539b3eedcc4ea4aa3566df9db5aa81: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 488 bytes on disk
I20240503 08:12:34.682293 27385 maintenance_manager.cc:643] P e4539b3eedcc4ea4aa3566df9db5aa81: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.720661 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.724357 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.742570 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.022s user 0.013s sys 0.000s Metrics: {"bytes_written":276542,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1229,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":650,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:12:34.742803 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1059756 bytes of WAL
I20240503 08:12:34.742898 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:34.742955 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000001 (ops 1-585)
I20240503 08:12:34.743381 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:34.743485 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 88431 bytes on disk
I20240503 08:12:34.743607 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.743690 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.743724 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014523
I20240503 08:12:34.751474 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.027s user 0.012s sys 0.002s Metrics: {"bytes_written":271207,"cfile_init":1,"dirs.queue_time_us":478,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":924,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1251,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20240503 08:12:34.751675 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1047864 bytes of WAL
I20240503 08:12:34.754662 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:34.754884 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000001 (ops 1-634)
I20240503 08:12:34.755906 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:34.756976 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014520
I20240503 08:12:34.778721 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.035s user 0.016s sys 0.000s Metrics: {"bytes_written":286938,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":827,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":708,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20240503 08:12:34.779484 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1061128 bytes of WAL
I20240503 08:12:34.779547 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:34.779567 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000001 (ops 1-631)
I20240503 08:12:34.780355 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:34.780643 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 92245 bytes on disk
I20240503 08:12:34.781704 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.781858 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014513
I20240503 08:12:34.795770 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.052s user 0.042s sys 0.004s Metrics: {"bytes_written":1084372,"cfile_cache_hit":12,"cfile_cache_hit_bytes":610378,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2117628,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":332,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":775,"drs_written":1,"lbm_read_time_us":1244,"lbm_reads_lt_1ms":82,"lbm_write_time_us":2654,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":577391,"rows_written":38200,"spinlock_wait_cycles":384}
I20240503 08:12:34.796308 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 382605 bytes on disk
I20240503 08:12:34.796439 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.849020 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.067s user 0.041s sys 0.003s Metrics: {"bytes_written":1110460,"cfile_cache_hit":12,"cfile_cache_hit_bytes":621758,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2177612,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1257,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":987,"drs_written":1,"lbm_read_time_us":3278,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":81,"lbm_write_time_us":4147,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":592641,"rows_written":39200,"spinlock_wait_cycles":384}
I20240503 08:12:34.849265 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 393264 bytes on disk
I20240503 08:12:34.849385 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.865151 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.108s user 0.047s sys 0.001s Metrics: {"bytes_written":1089542,"cfile_cache_hit":13,"cfile_cache_hit_bytes":638892,"cfile_cache_miss":61,"cfile_cache_miss_bytes":2103586,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":542,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1141,"drs_written":1,"lbm_read_time_us":8144,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":82,"lbm_write_time_us":1986,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":580441,"rows_written":38400}
I20240503 08:12:34.865374 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 384672 bytes on disk
I20240503 08:12:34.865497 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:34.916167 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.918102 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.931057 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.015s user 0.011s sys 0.001s Metrics: {"bytes_written":277881,"cfile_init":1,"dirs.queue_time_us":583,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":979,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":632,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:34.931231 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014282
I20240503 08:12:34.934116 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":285611,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":869,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9950,"spinlock_wait_cycles":280832}
I20240503 08:12:34.934370 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014283
I20240503 08:12:34.940737 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:34.970201 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.029s user 0.015s sys 0.000s Metrics: {"bytes_written":281616,"cfile_init":1,"dirs.queue_time_us":2040,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1125,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1936,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:34.970386 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014273
I20240503 08:12:35.020767 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.086s user 0.051s sys 0.006s Metrics: {"bytes_written":1363413,"cfile_cache_hit":11,"cfile_cache_hit_bytes":560078,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2888360,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":999,"drs_written":1,"lbm_read_time_us":2857,"lbm_reads_lt_1ms":95,"lbm_write_time_us":3826,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":729108,"rows_written":48150}
I20240503 08:12:35.021149 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 488447 bytes on disk
I20240503 08:12:35.021360 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.080430 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.149s user 0.050s sys 0.008s Metrics: {"bytes_written":1360823,"cfile_cache_hit":12,"cfile_cache_hit_bytes":585082,"cfile_cache_miss":70,"cfile_cache_miss_bytes":2856750,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":10833,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":1197,"drs_written":1,"lbm_read_time_us":1585,"lbm_reads_lt_1ms":94,"lbm_write_time_us":3532,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":727583,"rows_written":48050}
I20240503 08:12:35.080740 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 487300 bytes on disk
I20240503 08:12:35.080942 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.094097 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.122s user 0.051s sys 0.008s Metrics: {"bytes_written":1385150,"cfile_cache_hit":13,"cfile_cache_hit_bytes":703202,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2807872,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1467,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":811,"drs_written":1,"lbm_read_time_us":1641,"lbm_reads_lt_1ms":95,"lbm_write_time_us":7914,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":75,"num_input_rowsets":2,"peak_mem_usage":742071,"rows_written":49000}
I20240503 08:12:35.094398 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 497218 bytes on disk
I20240503 08:12:35.094595 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.111704 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.112589 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.126217 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.014s user 0.006s sys 0.004s Metrics: {"bytes_written":277879,"cfile_init":1,"dirs.queue_time_us":123,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1305,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":750,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650,"spinlock_wait_cycles":896}
I20240503 08:12:35.126525 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 88900 bytes on disk
I20240503 08:12:35.126713 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.014s user 0.008s sys 0.004s Metrics: {"bytes_written":275185,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":1335,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:35.126848 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.126953 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014328
I20240503 08:12:35.127025 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014328
I20240503 08:12:35.145682 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.164835 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.019s user 0.012s sys 0.003s Metrics: {"bytes_written":274979,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":999,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":897,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:35.165129 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 87979 bytes on disk
I20240503 08:12:35.165444 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.165592 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014319
I20240503 08:12:35.236872 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.110s user 0.069s sys 0.000s Metrics: {"bytes_written":1628732,"cfile_cache_hit":13,"cfile_cache_hit_bytes":745999,"cfile_cache_miss":81,"cfile_cache_miss_bytes":3394063,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":450,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":1124,"drs_written":1,"lbm_read_time_us":4124,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":104,"lbm_write_time_us":3201,"lbm_writes_lt_1ms":86,"num_input_rowsets":2,"peak_mem_usage":874723,"rows_written":57700,"spinlock_wait_cycles":384}
I20240503 08:12:35.237190 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 589365 bytes on disk
I20240503 08:12:35.237545 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.253903 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.088s user 0.067s sys 0.005s Metrics: {"bytes_written":1653745,"cfile_cache_hit":12,"cfile_cache_hit_bytes":719938,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3480820,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":706,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":1156,"drs_written":1,"lbm_read_time_us":2487,"lbm_reads_lt_1ms":102,"lbm_write_time_us":5847,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":87,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":887688,"rows_written":58550}
I20240503 08:12:35.254204 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 598539 bytes on disk
I20240503 08:12:35.254518 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.263459 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.136s user 0.068s sys 0.001s Metrics: {"bytes_written":1628732,"cfile_cache_hit":12,"cfile_cache_hit_bytes":611279,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3528631,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":341,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":5340,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":100,"lbm_write_time_us":5129,"lbm_writes_lt_1ms":86,"num_input_rowsets":2,"peak_mem_usage":874723,"rows_written":57700}
I20240503 08:12:35.263813 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 589365 bytes on disk
I20240503 08:12:35.264194 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.325592 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.329068 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.337328 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.012s user 0.006s sys 0.004s Metrics: {"bytes_written":280299,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":988,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:35.337606 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1051271 bytes of WAL
I20240503 08:12:35.337738 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:35.337810 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000002 (ops 635-1259)
I20240503 08:12:35.338304 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:35.338455 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 89873 bytes on disk
I20240503 08:12:35.338657 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.338814 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014234
I20240503 08:12:35.345790 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.348016 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.019s user 0.012s sys 0.000s Metrics: {"bytes_written":284362,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":1060,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":644,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20240503 08:12:35.348263 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1049736 bytes of WAL
I20240503 08:12:35.348366 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:35.348407 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000002 (ops 586-1163)
I20240503 08:12:35.349098 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:35.349225 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014233
I20240503 08:12:35.365942 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":273689,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":891,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:12:35.366179 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1050460 bytes of WAL
I20240503 08:12:35.366348 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:35.366429 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000002 (ops 632-1252)
I20240503 08:12:35.366884 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:35.366989 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014227
I20240503 08:12:35.466146 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.117s user 0.071s sys 0.005s Metrics: {"bytes_written":1908324,"cfile_cache_hit":11,"cfile_cache_hit_bytes":611003,"cfile_cache_miss":93,"cfile_cache_miss_bytes":4245509,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":229,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":960,"drs_written":1,"lbm_read_time_us":3784,"lbm_reads_lt_1ms":117,"lbm_write_time_us":4299,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1025685,"rows_written":67600,"spinlock_wait_cycles":768}
I20240503 08:12:35.466662 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 695010 bytes on disk
I20240503 08:12:35.466848 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.495440 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.157s user 0.071s sys 0.004s Metrics: {"bytes_written":1904094,"cfile_cache_hit":11,"cfile_cache_hit_bytes":608118,"cfile_cache_miss":93,"cfile_cache_miss_bytes":4237340,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":832,"drs_written":1,"lbm_read_time_us":2347,"lbm_reads_lt_1ms":113,"lbm_write_time_us":3955,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1023395,"rows_written":67450}
I20240503 08:12:35.495739 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 693628 bytes on disk
I20240503 08:12:35.495909 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.512668 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.143s user 0.072s sys 0.003s Metrics: {"bytes_written":1919514,"cfile_cache_hit":13,"cfile_cache_hit_bytes":798479,"cfile_cache_miss":93,"cfile_cache_miss_bytes":4089149,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":3678,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":991,"drs_written":1,"lbm_read_time_us":2533,"lbm_reads_lt_1ms":117,"lbm_write_time_us":3836,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1032543,"rows_written":68050,"spinlock_wait_cycles":896}
I20240503 08:12:35.512851 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 700065 bytes on disk
I20240503 08:12:35.512969 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.536535 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.537875 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.553655 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.556519 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":281654,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":921,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:35.556674 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 91941 bytes on disk
I20240503 08:12:35.556869 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.556962 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014107
I20240503 08:12:35.572078 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.034s user 0.015s sys 0.000s Metrics: {"bytes_written":280337,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":691,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1077,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:35.572284 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 91510 bytes on disk
I20240503 08:12:35.573599 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.573750 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014106
I20240503 08:12:35.578310 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.024s user 0.014s sys 0.000s Metrics: {"bytes_written":288136,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":818,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1045,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10050}
I20240503 08:12:35.578534 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 94122 bytes on disk
I20240503 08:12:35.578727 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.578868 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014099
I20240503 08:12:35.672816 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.116s user 0.083s sys 0.004s Metrics: {"bytes_written":2178325,"cfile_cache_hit":11,"cfile_cache_hit_bytes":478448,"cfile_cache_miss":107,"cfile_cache_miss_bytes":5083646,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1040,"drs_written":1,"lbm_read_time_us":2862,"lbm_reads_lt_1ms":127,"lbm_write_time_us":5112,"lbm_writes_lt_1ms":111,"num_input_rowsets":2,"peak_mem_usage":1172825,"rows_written":77250}
I20240503 08:12:35.673874 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 803290 bytes on disk
I20240503 08:12:35.674016 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.739012 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.159s user 0.080s sys 0.008s Metrics: {"bytes_written":2200106,"cfile_cache_hit":13,"cfile_cache_hit_bytes":696463,"cfile_cache_miss":105,"cfile_cache_miss_bytes":4926675,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":451,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":3410,"drs_written":1,"lbm_read_time_us":2774,"lbm_reads_lt_1ms":125,"lbm_write_time_us":8037,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":110,"num_input_rowsets":2,"peak_mem_usage":1186011,"rows_written":78100}
I20240503 08:12:35.739898 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 813373 bytes on disk
I20240503 08:12:35.740221 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.745575 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.751920 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.178s user 0.082s sys 0.007s Metrics: {"bytes_written":2180831,"cfile_cache_hit":13,"cfile_cache_hit_bytes":841100,"cfile_cache_miss":105,"cfile_cache_miss_bytes":4728730,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1499,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1067,"drs_written":1,"lbm_read_time_us":5075,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":123,"lbm_write_time_us":6209,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":110,"num_input_rowsets":2,"peak_mem_usage":1174345,"rows_written":77350}
I20240503 08:12:35.752130 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 804638 bytes on disk
I20240503 08:12:35.757838 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.758018 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.770771 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.774124 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.028s user 0.009s sys 0.004s Metrics: {"bytes_written":271170,"cfile_init":1,"dirs.queue_time_us":818,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":911,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3256,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9400}
I20240503 08:12:35.774286 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 86727 bytes on disk
I20240503 08:12:35.774449 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.774556 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014077
I20240503 08:12:35.778149 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":283108,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":739,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":645,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20240503 08:12:35.778329 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1053287 bytes of WAL
I20240503 08:12:35.778398 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:35.778424 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000003 (ops 1164-1743)
I20240503 08:12:35.778841 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:35.778928 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 90964 bytes on disk
I20240503 08:12:35.779027 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.779119 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014072
I20240503 08:12:35.796672 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.026s user 0.013s sys 0.002s Metrics: {"bytes_written":279062,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1031,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":743,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:12:35.796859 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 89668 bytes on disk
I20240503 08:12:35.796978 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.797058 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.014066
I20240503 08:12:35.903998 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.125s user 0.090s sys 0.006s Metrics: {"bytes_written":2451686,"cfile_cache_hit":11,"cfile_cache_hit_bytes":479688,"cfile_cache_miss":119,"cfile_cache_miss_bytes":5809438,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1050,"drs_written":1,"lbm_read_time_us":3814,"lbm_reads_lt_1ms":139,"lbm_write_time_us":5399,"lbm_writes_lt_1ms":122,"num_input_rowsets":2,"peak_mem_usage":1334071,"rows_written":87200}
I20240503 08:12:35.904583 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 916438 bytes on disk
I20240503 08:12:35.905933 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1174,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 08:12:35.951004 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.154s user 0.079s sys 0.010s Metrics: {"bytes_written":2470901,"cfile_cache_hit":13,"cfile_cache_hit_bytes":836112,"cfile_cache_miss":117,"cfile_cache_miss_bytes":5498342,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":229,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":4933,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":136,"lbm_write_time_us":6182,"lbm_writes_lt_1ms":123,"num_input_rowsets":2,"peak_mem_usage":1343976,"rows_written":87800,"spinlock_wait_cycles":1408}
I20240503 08:12:35.951333 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 923213 bytes on disk
I20240503 08:12:35.951494 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.957052 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.182s user 0.089s sys 0.005s Metrics: {"bytes_written":2438099,"cfile_cache_hit":13,"cfile_cache_hit_bytes":830071,"cfile_cache_miss":117,"cfile_cache_miss_bytes":5418977,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":302,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1131,"drs_written":1,"lbm_read_time_us":2770,"lbm_reads_lt_1ms":137,"lbm_write_time_us":7615,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":120,"num_input_rowsets":2,"peak_mem_usage":1325586,"rows_written":86650,"spinlock_wait_cycles":896}
I20240503 08:12:35.957458 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 909848 bytes on disk
I20240503 08:12:35.957633 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:35.957757 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.967556 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:35.971334 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":273725,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":1023,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500,"spinlock_wait_cycles":896}
I20240503 08:12:35.971534 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1056857 bytes of WAL
I20240503 08:12:35.971606 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:35.971639 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000003 (ops 1260-1891)
I20240503 08:12:35.972127 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:35.972262 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013697
I20240503 08:12:35.987568 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":272683,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":939,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1172,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:12:35.987804 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013692
I20240503 08:12:35.992112 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.021945 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.029s user 0.014s sys 0.000s Metrics: {"bytes_written":283169,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":1053,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2355,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9850}
I20240503 08:12:36.022223 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1064530 bytes of WAL
I20240503 08:12:36.022331 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:36.022390 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000003 (ops 1253-1883)
I20240503 08:12:36.023342 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:36.023851 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 90968 bytes on disk
I20240503 08:12:36.024039 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.024190 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013836
I20240503 08:12:36.076745 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.104s user 0.082s sys 0.016s Metrics: {"bytes_written":2706675,"cfile_cache_hit":10,"cfile_cache_hit_bytes":442183,"cfile_cache_miss":130,"cfile_cache_miss_bytes":6505577,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":1030,"drs_written":1,"lbm_read_time_us":3017,"lbm_reads_lt_1ms":154,"lbm_write_time_us":5307,"lbm_writes_lt_1ms":132,"num_input_rowsets":2,"peak_mem_usage":1479852,"rows_written":96150,"spinlock_wait_cycles":384}
I20240503 08:12:36.077060 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1017991 bytes on disk
I20240503 08:12:36.077250 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.156720 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.169s user 0.097s sys 0.010s Metrics: {"bytes_written":2720281,"cfile_cache_hit":12,"cfile_cache_hit_bytes":649862,"cfile_cache_miss":130,"cfile_cache_miss_bytes":6335170,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":250,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":1115,"drs_written":1,"lbm_read_time_us":4546,"lbm_reads_lt_1ms":154,"lbm_write_time_us":9915,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":130,"num_input_rowsets":2,"peak_mem_usage":1488021,"rows_written":96650,"spinlock_wait_cycles":2014336}
I20240503 08:12:36.156913 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1023623 bytes on disk
I20240503 08:12:36.157042 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.185601 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.161s user 0.083s sys 0.022s Metrics: {"bytes_written":2749038,"cfile_cache_hit":11,"cfile_cache_hit_bytes":627158,"cfile_cache_miss":131,"cfile_cache_miss_bytes":6429948,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1470,"drs_written":1,"lbm_read_time_us":3896,"lbm_reads_lt_1ms":151,"lbm_write_time_us":7777,"lbm_writes_lt_1ms":135,"mutex_wait_us":23,"num_input_rowsets":2,"peak_mem_usage":1503871,"rows_written":97650,"spinlock_wait_cycles":1280}
I20240503 08:12:36.185793 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1034688 bytes on disk
I20240503 08:12:36.185950 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.190443 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.204864 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":271195,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":964,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20240503 08:12:36.205113 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 86537 bytes on disk
I20240503 08:12:36.205288 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.205396 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013704
I20240503 08:12:36.208218 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.225548 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.017s user 0.012s sys 0.002s Metrics: {"bytes_written":283162,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1216,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20240503 08:12:36.225867 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013696
I20240503 08:12:36.228705 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.247220 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.018s user 0.011s sys 0.000s Metrics: {"bytes_written":272539,"cfile_init":1,"dirs.queue_time_us":92,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1191,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:12:36.248986 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 87074 bytes on disk
I20240503 08:12:36.249446 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":282,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.249635 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013659
I20240503 08:12:36.367507 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.162s user 0.103s sys 0.016s Metrics: {"bytes_written":2965777,"cfile_cache_hit":10,"cfile_cache_hit_bytes":440367,"cfile_cache_miss":140,"cfile_cache_miss_bytes":7203853,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":302,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":1463,"drs_written":1,"lbm_read_time_us":5431,"lbm_reads_lt_1ms":160,"lbm_write_time_us":7458,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":142,"num_input_rowsets":2,"peak_mem_usage":1632692,"rows_written":105550,"spinlock_wait_cycles":1280}
I20240503 08:12:36.367832 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1123089 bytes on disk
I20240503 08:12:36.368006 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.389066 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.139s user 0.109s sys 0.011s Metrics: {"bytes_written":3009669,"cfile_cache_hit":13,"cfile_cache_hit_bytes":644431,"cfile_cache_miss":141,"cfile_cache_miss_bytes":7113413,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1085,"drs_written":1,"lbm_read_time_us":5572,"lbm_reads_lt_1ms":161,"lbm_write_time_us":6534,"lbm_writes_lt_1ms":144,"num_input_rowsets":2,"peak_mem_usage":1657826,"rows_written":107100,"spinlock_wait_cycles":1536}
I20240503 08:12:36.389406 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1140063 bytes on disk
I20240503 08:12:36.389582 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.422549 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.196s user 0.111s sys 0.012s Metrics: {"bytes_written":2994289,"cfile_cache_hit":11,"cfile_cache_hit_bytes":626388,"cfile_cache_miss":139,"cfile_cache_miss_bytes":7088802,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":267,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1212,"drs_written":1,"lbm_read_time_us":7215,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":162,"lbm_write_time_us":8388,"lbm_writes_lt_1ms":144,"num_input_rowsets":2,"peak_mem_usage":1647677,"rows_written":106500}
I20240503 08:12:36.422768 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1133449 bytes on disk
I20240503 08:12:36.422889 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.449627 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.463362 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":281515,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1262,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:36.463754 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013682
I20240503 08:12:36.473595 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.491003 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.500699 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.027s user 0.014s sys 0.000s Metrics: {"bytes_written":277877,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":863,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":758,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:36.500886 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013673
I20240503 08:12:36.530709 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.039s user 0.013s sys 0.001s Metrics: {"bytes_written":285542,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":1220,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1623,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9950}
I20240503 08:12:36.531328 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013664
I20240503 08:12:36.639242 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.175s user 0.113s sys 0.012s Metrics: {"bytes_written":3239314,"cfile_cache_hit":11,"cfile_cache_hit_bytes":478680,"cfile_cache_miss":151,"cfile_cache_miss_bytes":7880980,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":1044,"drs_written":1,"lbm_read_time_us":4024,"lbm_reads_lt_1ms":175,"lbm_write_time_us":6674,"lbm_writes_lt_1ms":155,"num_input_rowsets":2,"peak_mem_usage":1791532,"rows_written":115350,"spinlock_wait_cycles":768}
I20240503 08:12:36.639549 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1232223 bytes on disk
I20240503 08:12:36.639734 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.651794 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.151s user 0.112s sys 0.004s Metrics: {"bytes_written":3259746,"cfile_cache_hit":13,"cfile_cache_hit_bytes":827106,"cfile_cache_miss":149,"cfile_cache_miss_bytes":7592550,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":961,"drs_written":1,"lbm_read_time_us":4006,"lbm_reads_lt_1ms":173,"lbm_write_time_us":8475,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":154,"num_input_rowsets":2,"peak_mem_usage":1804677,"rows_written":116150,"spinlock_wait_cycles":665984}
I20240503 08:12:36.652120 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1241505 bytes on disk
I20240503 08:12:36.652297 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.680408 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.692914 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.706717 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.026s user 0.015s sys 0.000s Metrics: {"bytes_written":280360,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":1019,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:36.707023 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 90090 bytes on disk
I20240503 08:12:36.707193 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.707352 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013591
I20240503 08:12:36.710973 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.018s user 0.012s sys 0.000s Metrics: {"bytes_written":279106,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":850,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:12:36.711117 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1069820 bytes of WAL
I20240503 08:12:36.711208 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:36.711241 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000004 (ops 1744-2332)
I20240503 08:12:36.711567 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:36.711647 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 89530 bytes on disk
I20240503 08:12:36.711771 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":28,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.711858 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013584
I20240503 08:12:36.752504 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.220s user 0.117s sys 0.010s Metrics: {"bytes_written":3287169,"cfile_cache_hit":13,"cfile_cache_hit_bytes":833413,"cfile_cache_miss":149,"cfile_cache_miss_bytes":7652005,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":736,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":982,"drs_written":1,"lbm_read_time_us":3836,"lbm_reads_lt_1ms":173,"lbm_write_time_us":6791,"lbm_writes_lt_1ms":156,"num_input_rowsets":2,"peak_mem_usage":1819002,"rows_written":117050,"spinlock_wait_cycles":1664}
I20240503 08:12:36.752880 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.781587 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.029s user 0.016s sys 0.000s Metrics: {"bytes_written":319864,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":877,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1610,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11100}
I20240503 08:12:36.781877 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1076762 bytes of WAL
I20240503 08:12:36.782871 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:36.782990 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000004 (ops 1884-2508)
I20240503 08:12:36.783447 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:36.783846 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013662
I20240503 08:12:36.925510 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.218s user 0.129s sys 0.006s Metrics: {"bytes_written":3513898,"cfile_cache_hit":11,"cfile_cache_hit_bytes":622574,"cfile_cache_miss":163,"cfile_cache_miss_bytes":8450198,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":924,"drs_written":1,"lbm_read_time_us":5149,"lbm_reads_lt_1ms":183,"lbm_write_time_us":7311,"lbm_writes_lt_1ms":166,"num_input_rowsets":2,"peak_mem_usage":1950002,"rows_written":125100,"thread_start_us":80,"threads_started":1}
I20240503 08:12:36.925858 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.936604 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.225s user 0.119s sys 0.016s Metrics: {"bytes_written":3532968,"cfile_cache_hit":11,"cfile_cache_hit_bytes":621382,"cfile_cache_miss":163,"cfile_cache_miss_bytes":8507186,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":935,"drs_written":1,"lbm_read_time_us":7292,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":182,"lbm_write_time_us":8491,"lbm_writes_lt_1ms":167,"num_input_rowsets":2,"peak_mem_usage":1962242,"rows_written":125850}
I20240503 08:12:36.936913 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1353414 bytes on disk
I20240503 08:12:36.937065 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:36.937206 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:36.951524 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.026s user 0.014s sys 0.003s Metrics: {"bytes_written":286835,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1121,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1124,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":163712}
I20240503 08:12:36.951802 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1046608 bytes of WAL
I20240503 08:12:36.951922 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:36.951980 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000004 (ops 1892-2526)
I20240503 08:12:36.952445 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:36.952574 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1436566 bytes on disk
I20240503 08:12:36.952744 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":44,"lbm_reads_lt_1ms":8}
I20240503 08:12:36.952847 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013381
I20240503 08:12:36.960603 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.023s user 0.014s sys 0.000s Metrics: {"bytes_written":276355,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":774,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1380,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:12:36.960938 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013417
I20240503 08:12:36.966243 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.182s user 0.129s sys 0.008s Metrics: {"bytes_written":3594029,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1133078,"cfile_cache_miss":161,"cfile_cache_miss_bytes":8163192,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":231,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":6272,"drs_written":1,"lbm_read_time_us":5217,"lbm_reads_lt_1ms":189,"lbm_write_time_us":6432,"lbm_writes_lt_1ms":169,"num_input_rowsets":2,"peak_mem_usage":1998210,"rows_written":128150,"spinlock_wait_cycles":1792}
I20240503 08:12:36.966496 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1380140 bytes on disk
I20240503 08:12:36.966792 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.007972 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.031085 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":280353,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":1134,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1840,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:37.031875 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 90056 bytes on disk
I20240503 08:12:37.032544 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":452,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.032799 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012929
I20240503 08:12:37.111814 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.159s user 0.127s sys 0.013s Metrics: {"bytes_written":3793070,"cfile_cache_hit":12,"cfile_cache_hit_bytes":784345,"cfile_cache_miss":172,"cfile_cache_miss_bytes":9029623,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":885,"drs_written":1,"lbm_read_time_us":5370,"lbm_reads_lt_1ms":192,"lbm_write_time_us":7585,"lbm_writes_lt_1ms":179,"num_input_rowsets":2,"peak_mem_usage":2112248,"rows_written":135100,"spinlock_wait_cycles":1408}
I20240503 08:12:37.112056 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1460072 bytes on disk
I20240503 08:12:37.112282 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.115038 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.154s user 0.121s sys 0.020s Metrics: {"bytes_written":3802517,"cfile_cache_hit":11,"cfile_cache_hit_bytes":605010,"cfile_cache_miss":175,"cfile_cache_miss_bytes":9236604,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":549,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":801,"drs_written":1,"lbm_read_time_us":4276,"lbm_reads_lt_1ms":199,"lbm_write_time_us":8127,"lbm_writes_lt_1ms":179,"num_input_rowsets":2,"peak_mem_usage":2118332,"rows_written":135450}
I20240503 08:12:37.115604 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1463624 bytes on disk
I20240503 08:12:37.115825 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.211884 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.179s user 0.128s sys 0.013s Metrics: {"bytes_written":3869510,"cfile_cache_hit":12,"cfile_cache_hit_bytes":779540,"cfile_cache_miss":176,"cfile_cache_miss_bytes":9240734,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":491,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1136,"drs_written":1,"lbm_read_time_us":4791,"lbm_reads_lt_1ms":196,"lbm_write_time_us":8366,"lbm_writes_lt_1ms":182,"num_input_rowsets":2,"peak_mem_usage":2157991,"rows_written":137900}
I20240503 08:12:37.212049 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1492314 bytes on disk
I20240503 08:12:37.212179 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.213686 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.217101 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.240355 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":275037,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1150,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:37.240865 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013096
I20240503 08:12:37.253104 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.039s user 0.015s sys 0.000s Metrics: {"bytes_written":277690,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":1031,"drs_written":1,"lbm_read_time_us":497,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1917,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:37.253305 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013125
I20240503 08:12:37.273380 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.294950 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.021s user 0.011s sys 0.003s Metrics: {"bytes_written":273747,"cfile_init":1,"dirs.queue_time_us":901,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":1287,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:12:37.295301 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013240
I20240503 08:12:37.400714 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.160s user 0.147s sys 0.009s Metrics: {"bytes_written":4064442,"cfile_cache_hit":12,"cfile_cache_hit_bytes":790643,"cfile_cache_miss":186,"cfile_cache_miss_bytes":9757815,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":1134,"drs_written":1,"lbm_read_time_us":4603,"lbm_reads_lt_1ms":210,"lbm_write_time_us":7873,"lbm_writes_lt_1ms":190,"num_input_rowsets":2,"peak_mem_usage":2273558,"rows_written":145000,"spinlock_wait_cycles":768}
I20240503 08:12:37.401049 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1572947 bytes on disk
I20240503 08:12:37.401347 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.477588 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.224s user 0.138s sys 0.014s Metrics: {"bytes_written":4058537,"cfile_cache_hit":10,"cfile_cache_hit_bytes":445659,"cfile_cache_miss":188,"cfile_cache_miss_bytes":10083997,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":901,"drs_written":1,"lbm_read_time_us":7598,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":211,"lbm_write_time_us":11340,"lbm_writes_lt_1ms":190,"num_input_rowsets":2,"peak_mem_usage":2269398,"rows_written":144750,"spinlock_wait_cycles":928128}
I20240503 08:12:37.477896 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1570010 bytes on disk
I20240503 08:12:37.478086 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.499068 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.502732 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.526933 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.028s user 0.014s sys 0.001s Metrics: {"bytes_written":280356,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1104,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:37.527204 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 89979 bytes on disk
I20240503 08:12:37.527789 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.527956 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013280
I20240503 08:12:37.540824 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.245s user 0.138s sys 0.020s Metrics: {"bytes_written":4135323,"cfile_cache_hit":12,"cfile_cache_hit_bytes":645664,"cfile_cache_miss":190,"cfile_cache_miss_bytes":10082296,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":913,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":768,"drs_written":1,"lbm_read_time_us":5828,"lbm_reads_lt_1ms":214,"lbm_write_time_us":13634,"lbm_writes_lt_1ms":193,"num_input_rowsets":2,"peak_mem_usage":2312606,"rows_written":147400}
I20240503 08:12:37.542398 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1599464 bytes on disk
I20240503 08:12:37.542995 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.545791 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.041s user 0.012s sys 0.002s Metrics: {"bytes_written":281554,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":1196,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1618,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:37.545964 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1066210 bytes of WAL
I20240503 08:12:37.546150 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:37.546200 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000005 (ops 2333-2919)
I20240503 08:12:37.546762 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:37.547013 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013277
I20240503 08:12:37.574559 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.604120 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.029s user 0.014s sys 0.000s Metrics: {"bytes_written":281589,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":840,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:37.604339 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1050273 bytes of WAL
I20240503 08:12:37.604468 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:37.604502 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000005 (ops 2509-3125)
I20240503 08:12:37.604868 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:37.606096 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013145
I20240503 08:12:37.715294 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.187s user 0.150s sys 0.011s Metrics: {"bytes_written":4331597,"cfile_cache_hit":11,"cfile_cache_hit_bytes":478195,"cfile_cache_miss":199,"cfile_cache_miss_bytes":10768291,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":1024,"drs_written":1,"lbm_read_time_us":5731,"lbm_reads_lt_1ms":219,"lbm_write_time_us":8373,"lbm_writes_lt_1ms":202,"num_input_rowsets":2,"peak_mem_usage":2427723,"rows_written":154500,"spinlock_wait_cycles":1152}
I20240503 08:12:37.717639 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1680487 bytes on disk
I20240503 08:12:37.718269 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.739053 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.192s user 0.136s sys 0.022s Metrics: {"bytes_written":4338779,"cfile_cache_hit":12,"cfile_cache_hit_bytes":794067,"cfile_cache_miss":198,"cfile_cache_miss_bytes":10473435,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":929,"drs_written":1,"lbm_read_time_us":4962,"lbm_reads_lt_1ms":222,"lbm_write_time_us":19297,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":201,"num_input_rowsets":2,"peak_mem_usage":2432548,"rows_written":154800,"spinlock_wait_cycles":384}
I20240503 08:12:37.740234 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1683764 bytes on disk
I20240503 08:12:37.740490 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.769747 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.781100 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.812122 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.042s user 0.011s sys 0.004s Metrics: {"bytes_written":274959,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":733,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":852,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:37.812480 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1059651 bytes of WAL
I20240503 08:12:37.812621 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:37.812873 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000005 (ops 2527-3140)
I20240503 08:12:37.813663 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:37.813797 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 88135 bytes on disk
I20240503 08:12:37.813943 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.814069 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013166
I20240503 08:12:37.822254 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.041s user 0.015s sys 0.000s Metrics: {"bytes_written":280244,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":798,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:37.822496 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 89871 bytes on disk
I20240503 08:12:37.823333 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.823498 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013161
I20240503 08:12:37.836076 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.229s user 0.151s sys 0.009s Metrics: {"bytes_written":4403115,"cfile_cache_hit":14,"cfile_cache_hit_bytes":997577,"cfile_cache_miss":198,"cfile_cache_miss_bytes":10450357,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":1051,"drs_written":1,"lbm_read_time_us":5898,"lbm_reads_lt_1ms":222,"lbm_write_time_us":11312,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":203,"num_input_rowsets":2,"peak_mem_usage":2471547,"rows_written":157200,"spinlock_wait_cycles":896}
I20240503 08:12:37.836388 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1711654 bytes on disk
I20240503 08:12:37.836611 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:37.836767 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:37.877148 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.040s user 0.014s sys 0.000s Metrics: {"bytes_written":273785,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":889,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:12:37.877552 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013226
I20240503 08:12:38.072211 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.258s user 0.163s sys 0.009s Metrics: {"bytes_written":4595980,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1131771,"cfile_cache_miss":208,"cfile_cache_miss_bytes":10817895,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":716,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":8389,"lbm_reads_lt_1ms":228,"lbm_write_time_us":12318,"lbm_writes_lt_1ms":212,"num_input_rowsets":2,"peak_mem_usage":2583098,"rows_written":164050,"spinlock_wait_cycles":22912}
I20240503 08:12:38.072535 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1791451 bytes on disk
I20240503 08:12:38.072912 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.073062 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.074262 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.251s user 0.155s sys 0.017s Metrics: {"bytes_written":4613191,"cfile_cache_hit":14,"cfile_cache_hit_bytes":993712,"cfile_cache_miss":208,"cfile_cache_miss_bytes":10990888,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":215,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":938,"drs_written":1,"lbm_read_time_us":8938,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":227,"lbm_write_time_us":9849,"lbm_writes_lt_1ms":213,"num_input_rowsets":2,"peak_mem_usage":2591023,"rows_written":164550,"spinlock_wait_cycles":125568}
I20240503 08:12:38.074581 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1797107 bytes on disk
I20240503 08:12:38.074752 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.074884 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.089051 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":317614,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11000,"spinlock_wait_cycles":896}
I20240503 08:12:38.089298 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 101911 bytes on disk
I20240503 08:12:38.089439 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.089550 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013094
I20240503 08:12:38.109831 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.035s user 0.022s sys 0.000s Metrics: {"bytes_written":305555,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":1051,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10550}
I20240503 08:12:38.110037 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1055263 bytes of WAL
I20240503 08:12:38.110137 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:38.110185 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000006 (ops 2920-3500)
I20240503 08:12:38.110944 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:38.111315 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 97425 bytes on disk
I20240503 08:12:38.112241 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":240,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.112653 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013092
I20240503 08:12:38.167896 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.290s user 0.161s sys 0.009s Metrics: {"bytes_written":4671784,"cfile_cache_hit":13,"cfile_cache_hit_bytes":751614,"cfile_cache_miss":211,"cfile_cache_miss_bytes":11391690,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":250,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":809,"drs_written":1,"lbm_read_time_us":9313,"lbm_reads_lt_1ms":235,"lbm_write_time_us":15807,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":212,"num_input_rowsets":2,"peak_mem_usage":2626212,"rows_written":166700,"spinlock_wait_cycles":768}
I20240503 08:12:38.168264 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.190901 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.022s user 0.015s sys 0.001s Metrics: {"bytes_written":324172,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":2397,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1352,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11250}
I20240503 08:12:38.191308 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1926570 bytes on disk
I20240503 08:12:38.191519 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":37,"lbm_reads_lt_1ms":8}
I20240503 08:12:38.191732 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.013063
I20240503 08:12:38.320178 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.205s user 0.168s sys 0.017s Metrics: {"bytes_written":4904490,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1042515,"cfile_cache_miss":217,"cfile_cache_miss_bytes":11725669,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1037,"drs_written":1,"lbm_read_time_us":5556,"lbm_reads_lt_1ms":237,"lbm_write_time_us":9343,"lbm_writes_lt_1ms":225,"num_input_rowsets":2,"peak_mem_usage":2761648,"rows_written":175100,"spinlock_wait_cycles":384}
I20240503 08:12:38.320809 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1919647 bytes on disk
I20240503 08:12:38.322119 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.373744 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.389427 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.016s user 0.007s sys 0.006s Metrics: {"bytes_written":279211,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":945,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:12:38.389746 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012840
I20240503 08:12:38.393405 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.202s user 0.158s sys 0.021s Metrics: {"bytes_written":4985641,"cfile_cache_hit":12,"cfile_cache_hit_bytes":663325,"cfile_cache_miss":226,"cfile_cache_miss_bytes":12315127,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":263,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":1274,"drs_written":1,"lbm_read_time_us":7093,"lbm_reads_lt_1ms":246,"lbm_write_time_us":10623,"lbm_writes_lt_1ms":229,"num_input_rowsets":2,"peak_mem_usage":2807260,"rows_written":177950,"spinlock_wait_cycles":896}
I20240503 08:12:38.393627 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 1952059 bytes on disk
I20240503 08:12:38.393930 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.432303 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.343s user 0.168s sys 0.017s Metrics: {"bytes_written":4903246,"cfile_cache_hit":12,"cfile_cache_hit_bytes":677565,"cfile_cache_miss":222,"cfile_cache_miss_bytes":12086215,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":1133,"drs_written":1,"lbm_read_time_us":6370,"lbm_reads_lt_1ms":242,"lbm_write_time_us":21121,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":220,"num_input_rowsets":2,"peak_mem_usage":2760381,"rows_written":175050,"spinlock_wait_cycles":1408}
I20240503 08:12:38.432672 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.445611 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.455206 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.022s user 0.013s sys 0.003s Metrics: {"bytes_written":342442,"cfile_init":1,"dirs.queue_time_us":31,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11950}
I20240503 08:12:38.455456 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2030280 bytes on disk
I20240503 08:12:38.455673 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":37,"lbm_reads_lt_1ms":8}
I20240503 08:12:38.455832 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012820
I20240503 08:12:38.484033 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.038s user 0.016s sys 0.000s Metrics: {"bytes_written":273740,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":924,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1336,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:12:38.484285 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1065087 bytes of WAL
I20240503 08:12:38.485095 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:38.485183 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000006 (ops 3126-3749)
I20240503 08:12:38.486405 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:38.486549 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012811
I20240503 08:12:38.580590 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.191s user 0.172s sys 0.016s Metrics: {"bytes_written":5173534,"cfile_cache_hit":12,"cfile_cache_hit_bytes":676228,"cfile_cache_miss":232,"cfile_cache_miss_bytes":12809892,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1261,"drs_written":1,"lbm_read_time_us":5841,"lbm_reads_lt_1ms":256,"lbm_write_time_us":9154,"lbm_writes_lt_1ms":237,"num_input_rowsets":2,"peak_mem_usage":2920118,"rows_written":184800}
I20240503 08:12:38.580905 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2030759 bytes on disk
I20240503 08:12:38.581089 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.661731 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.206s user 0.169s sys 0.024s Metrics: {"bytes_written":5234171,"cfile_cache_hit":11,"cfile_cache_hit_bytes":527298,"cfile_cache_miss":235,"cfile_cache_miss_bytes":13116840,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":1066,"drs_written":1,"lbm_read_time_us":6304,"lbm_reads_lt_1ms":255,"lbm_write_time_us":9419,"lbm_writes_lt_1ms":239,"num_input_rowsets":2,"peak_mem_usage":2953611,"rows_written":187000,"spinlock_wait_cycles":2048}
I20240503 08:12:38.662073 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2056101 bytes on disk
I20240503 08:12:38.662269 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.664772 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.697526 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":280351,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1053,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1928,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:38.698091 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012890
I20240503 08:12:38.723135 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.786641 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.063s user 0.014s sys 0.003s Metrics: {"bytes_written":277843,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11342,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:38.786959 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1058685 bytes of WAL
I20240503 08:12:38.787545 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.300s user 0.188s sys 0.011s Metrics: {"bytes_written":5246054,"cfile_cache_hit":13,"cfile_cache_hit_bytes":857280,"cfile_cache_miss":235,"cfile_cache_miss_bytes":12824684,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":908,"drs_written":1,"lbm_read_time_us":13322,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":257,"lbm_write_time_us":13731,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":238,"num_input_rowsets":2,"peak_mem_usage":2963376,"rows_written":187450,"spinlock_wait_cycles":230144}
I20240503 08:12:38.787907 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2060811 bytes on disk
I20240503 08:12:38.788074 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.788866 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.787698 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:38.790680 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000006 (ops 3141-3769)
I20240503 08:12:38.791142 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:38.791278 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012868
I20240503 08:12:38.809540 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.009s sys 0.008s Metrics: {"bytes_written":324075,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":981,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":930,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11250,"spinlock_wait_cycles":768}
I20240503 08:12:38.809963 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012848
I20240503 08:12:38.947448 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.249s user 0.179s sys 0.020s Metrics: {"bytes_written":5443036,"cfile_cache_hit":11,"cfile_cache_hit_bytes":478646,"cfile_cache_miss":245,"cfile_cache_miss_bytes":13724592,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":941,"drs_written":1,"lbm_read_time_us":8024,"lbm_reads_lt_1ms":269,"lbm_write_time_us":12353,"lbm_writes_lt_1ms":248,"num_input_rowsets":2,"peak_mem_usage":3078488,"rows_written":194550,"spinlock_wait_cycles":896}
I20240503 08:12:38.948266 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2142954 bytes on disk
I20240503 08:12:38.948504 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:38.948907 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:38.966358 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":284335,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1553,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20240503 08:12:38.966732 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1057143 bytes of WAL
I20240503 08:12:38.967041 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:38.967085 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000007 (ops 3501-4082)
I20240503 08:12:38.972025 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:38.972215 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012730
I20240503 08:12:39.050731 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.241s user 0.197s sys 0.013s Metrics: {"bytes_written":5558665,"cfile_cache_hit":12,"cfile_cache_hit_bytes":681090,"cfile_cache_miss":248,"cfile_cache_miss_bytes":13827206,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2459,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":11831,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":270,"lbm_write_time_us":12252,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":254,"num_input_rowsets":2,"peak_mem_usage":3144425,"rows_written":198700,"spinlock_wait_cycles":2688}
I20240503 08:12:39.051071 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2191722 bytes on disk
I20240503 08:12:39.051328 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:39.051584 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.076385 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.024s user 0.013s sys 0.002s Metrics: {"bytes_written":284362,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":860,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":996,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20240503 08:12:39.076813 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 91230 bytes on disk
I20240503 08:12:39.077010 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:39.077222 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012733
I20240503 08:12:39.180265 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.389s user 0.187s sys 0.016s Metrics: {"bytes_written":5499663,"cfile_cache_hit":12,"cfile_cache_hit_bytes":645906,"cfile_cache_miss":246,"cfile_cache_miss_bytes":13713846,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1183,"drs_written":1,"lbm_read_time_us":12331,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":268,"lbm_write_time_us":19289,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":248,"num_input_rowsets":2,"peak_mem_usage":3112712,"rows_written":196650,"thread_start_us":202,"threads_started":5}
I20240503 08:12:39.180665 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.214525 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.034s user 0.023s sys 0.000s Metrics: {"bytes_written":472797,"cfile_init":1,"dirs.queue_time_us":205,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":823,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1064,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16550}
I20240503 08:12:39.214869 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012690
I20240503 08:12:39.290494 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.318s user 0.195s sys 0.015s Metrics: {"bytes_written":5719132,"cfile_cache_hit":12,"cfile_cache_hit_bytes":650729,"cfile_cache_miss":256,"cfile_cache_miss_bytes":14282131,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1093,"drs_written":1,"lbm_read_time_us":11987,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":279,"lbm_write_time_us":15281,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":260,"num_input_rowsets":2,"peak_mem_usage":3239208,"rows_written":204450}
I20240503 08:12:39.290846 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.311996 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.021s user 0.014s sys 0.004s Metrics: {"bytes_written":355959,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":902,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":912,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12300}
I20240503 08:12:39.312307 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012696
I20240503 08:12:39.409662 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.332s user 0.202s sys 0.017s Metrics: {"bytes_written":5833164,"cfile_cache_hit":10,"cfile_cache_hit_bytes":451249,"cfile_cache_miss":266,"cfile_cache_miss_bytes":14785803,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":870,"drs_written":1,"lbm_read_time_us":11055,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":284,"lbm_write_time_us":14439,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":264,"num_input_rowsets":2,"peak_mem_usage":3306641,"rows_written":208600}
I20240503 08:12:39.410001 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.448781 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.036s user 0.013s sys 0.005s Metrics: {"bytes_written":361210,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":1122,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":932,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12500}
I20240503 08:12:39.449082 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1057686 bytes of WAL
I20240503 08:12:39.449275 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:39.449352 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000007 (ops 3750-4371)
I20240503 08:12:39.449817 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:39.449962 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012651
I20240503 08:12:39.532464 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.317s user 0.187s sys 0.034s Metrics: {"bytes_written":5960563,"cfile_cache_hit":13,"cfile_cache_hit_bytes":1034026,"cfile_cache_miss":269,"cfile_cache_miss_bytes":14527646,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":839,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1087,"drs_written":1,"lbm_read_time_us":9825,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":296,"lbm_write_time_us":13778,"lbm_writes_lt_1ms":271,"num_input_rowsets":2,"peak_mem_usage":3374726,"rows_written":213200,"spinlock_wait_cycles":384}
I20240503 08:12:39.532759 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.558704 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.026s user 0.013s sys 0.004s Metrics: {"bytes_written":333055,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":886,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11600}
I20240503 08:12:39.559258 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1052955 bytes of WAL
I20240503 08:12:39.559499 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:39.559549 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000007 (ops 3770-4417)
I20240503 08:12:39.560668 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:39.561172 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012631
I20240503 08:12:39.595755 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.283s user 0.193s sys 0.031s Metrics: {"bytes_written":6058750,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1039931,"cfile_cache_miss":270,"cfile_cache_miss_bytes":14800433,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":773,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":10714,"lbm_reads_lt_1ms":298,"lbm_write_time_us":13339,"lbm_writes_lt_1ms":275,"num_input_rowsets":2,"peak_mem_usage":3436661,"rows_written":216750,"spinlock_wait_cycles":1152}
I20240503 08:12:39.596125 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.621301 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"bytes_written":288159,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":969,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":772,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10050}
I20240503 08:12:39.621605 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012551
I20240503 08:12:39.821048 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.371s user 0.202s sys 0.026s Metrics: {"bytes_written":6183247,"cfile_cache_hit":12,"cfile_cache_hit_bytes":562990,"cfile_cache_miss":278,"cfile_cache_miss_bytes":15598286,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":882,"drs_written":1,"lbm_read_time_us":11850,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":304,"lbm_write_time_us":17799,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":278,"num_input_rowsets":2,"peak_mem_usage":3507144,"rows_written":221100,"spinlock_wait_cycles":1152}
I20240503 08:12:39.821293 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2450049 bytes on disk
I20240503 08:12:39.822050 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:39.822216 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.858435 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.036s user 0.014s sys 0.004s Metrics: {"bytes_written":378392,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":855,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1460,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13150}
I20240503 08:12:39.858764 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012472
I20240503 08:12:39.920445 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.359s user 0.218s sys 0.021s Metrics: {"bytes_written":6282901,"cfile_cache_hit":13,"cfile_cache_hit_bytes":856470,"cfile_cache_miss":281,"cfile_cache_miss_bytes":15575844,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":988,"drs_written":1,"lbm_read_time_us":10692,"lbm_reads_lt_1ms":309,"lbm_write_time_us":19356,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":284,"num_input_rowsets":2,"peak_mem_usage":3568167,"rows_written":224800,"spinlock_wait_cycles":2048}
I20240503 08:12:39.920643 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:39.950856 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.030s user 0.019s sys 0.000s Metrics: {"bytes_written":375789,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":1088,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1175,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":13050}
I20240503 08:12:39.951042 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012483
I20240503 08:12:39.989221 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.367s user 0.221s sys 0.021s Metrics: {"bytes_written":6337823,"cfile_cache_hit":14,"cfile_cache_hit_bytes":988043,"cfile_cache_miss":282,"cfile_cache_miss_bytes":15594343,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":999,"drs_written":1,"lbm_read_time_us":12042,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":309,"lbm_write_time_us":19290,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":286,"num_input_rowsets":2,"peak_mem_usage":3602217,"rows_written":226800,"spinlock_wait_cycles":384}
I20240503 08:12:39.989396 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2515336 bytes on disk
I20240503 08:12:39.989511 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:39.989643 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.039865 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.050s user 0.018s sys 0.004s Metrics: {"bytes_written":386250,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":906,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3367,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":13450}
I20240503 08:12:40.040066 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1051531 bytes of WAL
I20240503 08:12:40.041206 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:40.041343 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000008 (ops 4083-4661)
I20240503 08:12:40.042678 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:40.042790 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012458
I20240503 08:12:40.252147 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.393s user 0.223s sys 0.021s Metrics: {"bytes_written":6545815,"cfile_cache_hit":11,"cfile_cache_hit_bytes":681425,"cfile_cache_miss":293,"cfile_cache_miss_bytes":16453181,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":968,"drs_written":1,"lbm_read_time_us":12735,"lbm_reads_lt_1ms":317,"lbm_write_time_us":20426,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":293,"num_input_rowsets":2,"peak_mem_usage":3720166,"rows_written":234250,"spinlock_wait_cycles":2048}
I20240503 08:12:40.252377 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.320549 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.369s user 0.214s sys 0.031s Metrics: {"bytes_written":6647518,"cfile_cache_hit":14,"cfile_cache_hit_bytes":862817,"cfile_cache_miss":294,"cfile_cache_miss_bytes":16529505,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":9655,"drs_written":1,"lbm_read_time_us":13689,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":321,"lbm_write_time_us":19282,"lbm_writes_lt_1ms":300,"num_input_rowsets":2,"peak_mem_usage":3778766,"rows_written":237850,"spinlock_wait_cycles":768}
I20240503 08:12:40.321367 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2643157 bytes on disk
I20240503 08:12:40.321537 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.321722 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.336225 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.084s user 0.018s sys 0.004s Metrics: {"bytes_written":412172,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":838,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1780,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14300}
I20240503 08:12:40.336510 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1060729 bytes of WAL
I20240503 08:12:40.336658 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:40.336737 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000008 (ops 4372-4969)
I20240503 08:12:40.339464 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:40.339727 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012375
I20240503 08:12:40.346094 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.024s user 0.013s sys 0.005s Metrics: {"bytes_written":371927,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":902,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":825,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12900}
I20240503 08:12:40.346735 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 120113 bytes on disk
I20240503 08:12:40.347138 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.349689 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012304
I20240503 08:12:40.409701 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.366s user 0.213s sys 0.040s Metrics: {"bytes_written":6711316,"cfile_cache_hit":11,"cfile_cache_hit_bytes":561179,"cfile_cache_miss":301,"cfile_cache_miss_bytes":17006251,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":497,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":993,"drs_written":1,"lbm_read_time_us":10442,"lbm_reads_lt_1ms":325,"lbm_write_time_us":18198,"lbm_writes_lt_1ms":302,"mutex_wait_us":2,"num_input_rowsets":2,"peak_mem_usage":3817358,"rows_written":240250,"spinlock_wait_cycles":2392192}
I20240503 08:12:40.410018 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2671619 bytes on disk
I20240503 08:12:40.410192 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.410321 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.435196 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.019s sys 0.000s Metrics: {"bytes_written":403295,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":886,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1071,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":13950}
I20240503 08:12:40.435467 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 130595 bytes on disk
I20240503 08:12:40.435648 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.436208 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012268
I20240503 08:12:40.644311 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.294s user 0.239s sys 0.016s Metrics: {"bytes_written":7003531,"cfile_cache_hit":12,"cfile_cache_hit_bytes":842734,"cfile_cache_miss":310,"cfile_cache_miss_bytes":17501802,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":908,"drs_written":1,"lbm_read_time_us":14155,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":329,"lbm_write_time_us":13911,"lbm_writes_lt_1ms":315,"num_input_rowsets":2,"peak_mem_usage":3988525,"rows_written":250750,"spinlock_wait_cycles":2129792}
I20240503 08:12:40.644659 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2793127 bytes on disk
I20240503 08:12:40.644874 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.665328 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.692063 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.014s sys 0.001s Metrics: {"bytes_written":277707,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":907,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1032,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:40.692332 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1049113 bytes of WAL
I20240503 08:12:40.692449 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:40.692516 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000008 (ops 4418-5021)
I20240503 08:12:40.693495 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:40.693640 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011708
I20240503 08:12:40.742365 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.306s user 0.249s sys 0.020s Metrics: {"bytes_written":7098464,"cfile_cache_hit":11,"cfile_cache_hit_bytes":693925,"cfile_cache_miss":317,"cfile_cache_miss_bytes":17902373,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":805,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":875,"drs_written":1,"lbm_read_time_us":9864,"lbm_reads_lt_1ms":337,"lbm_write_time_us":14648,"lbm_writes_lt_1ms":318,"num_input_rowsets":2,"peak_mem_usage":4043532,"rows_written":254200,"spinlock_wait_cycles":1920}
I20240503 08:12:40.743809 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2832468 bytes on disk
I20240503 08:12:40.744025 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.785975 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.799211 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.457s user 0.248s sys 0.015s Metrics: {"bytes_written":6944434,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1238168,"cfile_cache_miss":305,"cfile_cache_miss_bytes":16944174,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":911,"drs_written":1,"lbm_read_time_us":10403,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":332,"lbm_write_time_us":15330,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":311,"num_input_rowsets":2,"peak_mem_usage":3951380,"rows_written":248550,"spinlock_wait_cycles":2304,"thread_start_us":205,"threads_started":5}
I20240503 08:12:40.799638 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:40.801865 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.016s user 0.010s sys 0.003s Metrics: {"bytes_written":271127,"cfile_init":1,"dirs.queue_time_us":103,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":1088,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":682,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20240503 08:12:40.802119 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1055277 bytes of WAL
I20240503 08:12:40.802358 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:40.802448 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000009 (ops 4662-5242)
I20240503 08:12:40.802948 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:40.804158 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 87950 bytes on disk
I20240503 08:12:40.804620 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":29,"lbm_reads_lt_1ms":4}
I20240503 08:12:40.804807 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011650
I20240503 08:12:40.832692 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.033s user 0.018s sys 0.006s Metrics: {"bytes_written":450272,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":2317,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2450,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":15650}
I20240503 08:12:40.833001 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 2916924 bytes on disk
I20240503 08:12:40.833243 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":35,"lbm_reads_lt_1ms":8}
I20240503 08:12:40.833834 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011903
I20240503 08:12:41.317981 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.484s user 0.293s sys 0.016s Metrics: {"bytes_written":7377931,"cfile_cache_hit":13,"cfile_cache_hit_bytes":907173,"cfile_cache_miss":327,"cfile_cache_miss_bytes":18429317,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":107,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1034,"drs_written":1,"lbm_read_time_us":13173,"lbm_reads_lt_1ms":347,"lbm_write_time_us":24801,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":330,"num_input_rowsets":2,"peak_mem_usage":4204320,"rows_written":264200,"spinlock_wait_cycles":2048}
I20240503 08:12:41.319121 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:41.320240 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.626s user 0.228s sys 0.046s Metrics: {"bytes_written":7273730,"cfile_cache_hit":12,"cfile_cache_hit_bytes":642844,"cfile_cache_miss":324,"cfile_cache_miss_bytes":18421904,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":135,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1051,"drs_written":1,"lbm_read_time_us":21687,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":345,"lbm_write_time_us":34212,"lbm_writes_1-10_ms":10,"lbm_writes_lt_1ms":315,"num_input_rowsets":2,"peak_mem_usage":4148580,"rows_written":260400,"spinlock_wait_cycles":1920,"thread_start_us":329,"threads_started":6}
I20240503 08:12:41.320504 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:41.353055 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.034s user 0.021s sys 0.004s Metrics: {"bytes_written":569914,"cfile_init":1,"dirs.queue_time_us":865,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2325,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":20000}
I20240503 08:12:41.353710 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1065437 bytes of WAL
I20240503 08:12:41.354235 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:41.354321 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000009 (ops 4970-5586)
I20240503 08:12:41.355105 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:41.355358 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3139533 bytes on disk
I20240503 08:12:41.355876 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":42,"lbm_reads_lt_1ms":8}
I20240503 08:12:41.356380 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012005
I20240503 08:12:41.417120 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.095s user 0.033s sys 0.000s Metrics: {"bytes_written":676813,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":913,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2833,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":23800}
I20240503 08:12:41.417521 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1057762 bytes of WAL
I20240503 08:12:41.417613 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:41.417690 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000009 (ops 5022-5673)
I20240503 08:12:41.418502 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:41.418673 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3132292 bytes on disk
I20240503 08:12:41.418911 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":40,"lbm_reads_lt_1ms":8}
I20240503 08:12:41.419214 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.012007
I20240503 08:12:41.466125 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.660s user 0.247s sys 0.030s Metrics: {"bytes_written":7363234,"cfile_cache_hit":14,"cfile_cache_hit_bytes":969468,"cfile_cache_miss":324,"cfile_cache_miss_bytes":18332030,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":13905,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":2053,"drs_written":1,"lbm_read_time_us":17160,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":341,"lbm_write_time_us":31611,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":328,"num_input_rowsets":2,"peak_mem_usage":4200829,"rows_written":263600,"spinlock_wait_cycles":2432,"thread_start_us":393,"threads_started":6}
I20240503 08:12:41.466475 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:41.517593 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.051s user 0.028s sys 0.004s Metrics: {"bytes_written":771144,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":100,"dirs.run_wall_time_us":1340,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3253,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":50,"peak_mem_usage":0,"rows_written":27100}
I20240503 08:12:41.517901 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3204284 bytes on disk
I20240503 08:12:41.518229 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":41,"lbm_reads_lt_1ms":8}
I20240503 08:12:41.518427 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011938
I20240503 08:12:41.813625 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.394s user 0.268s sys 0.022s Metrics: {"bytes_written":7933026,"cfile_cache_hit":13,"cfile_cache_hit_bytes":921462,"cfile_cache_miss":345,"cfile_cache_miss_bytes":19866806,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":841,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":909,"drs_written":1,"lbm_read_time_us":10344,"lbm_reads_lt_1ms":365,"lbm_write_time_us":24910,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":352,"num_input_rowsets":2,"peak_mem_usage":4521144,"rows_written":284200,"spinlock_wait_cycles":1572608}
I20240503 08:12:41.814364 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:41.848948 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.492s user 0.249s sys 0.041s Metrics: {"bytes_written":7933026,"cfile_cache_hit":12,"cfile_cache_hit_bytes":753519,"cfile_cache_miss":354,"cfile_cache_miss_bytes":20043805,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":10964,"lbm_reads_lt_1ms":374,"lbm_write_time_us":52987,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":351,"num_input_rowsets":2,"peak_mem_usage":4524944,"rows_written":284200,"spinlock_wait_cycles":1198336}
I20240503 08:12:41.849532 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3180110 bytes on disk
I20240503 08:12:41.849857 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:41.850085 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:41.873077 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.056s user 0.024s sys 0.003s Metrics: {"bytes_written":565973,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":5207,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3929,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":19850}
I20240503 08:12:41.873396 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3370971 bytes on disk
I20240503 08:12:41.873618 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":35,"lbm_reads_lt_1ms":8}
I20240503 08:12:41.874056 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011727
I20240503 08:12:41.882586 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.032s user 0.027s sys 0.000s Metrics: {"bytes_written":602651,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":658,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2533,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":21100}
I20240503 08:12:41.882879 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011714
I20240503 08:12:42.032325 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.513s user 0.283s sys 0.015s Metrics: {"bytes_written":8111849,"cfile_cache_hit":12,"cfile_cache_hit_bytes":772558,"cfile_cache_miss":360,"cfile_cache_miss_bytes":20494382,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1079,"drs_written":1,"lbm_read_time_us":21528,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":379,"lbm_write_time_us":26551,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":360,"num_input_rowsets":2,"peak_mem_usage":4623458,"rows_written":290700,"spinlock_wait_cycles":2560,"thread_start_us":212,"threads_started":5}
I20240503 08:12:42.032706 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3254693 bytes on disk
I20240503 08:12:42.032924 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:42.033119 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:42.071630 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.038s user 0.026s sys 0.001s Metrics: {"bytes_written":543729,"cfile_init":1,"dirs.queue_time_us":16,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":704,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1776,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":19000}
I20240503 08:12:42.072069 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1084195 bytes of WAL
I20240503 08:12:42.072256 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:42.072952 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000010 (ops 5243-5839)
I20240503 08:12:42.073491 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:42.073696 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011685
I20240503 08:12:42.231870 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.349s user 0.281s sys 0.033s Metrics: {"bytes_written":8518294,"cfile_cache_hit":13,"cfile_cache_hit_bytes":920821,"cfile_cache_miss":375,"cfile_cache_miss_bytes":21433761,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":498,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1039,"drs_written":1,"lbm_read_time_us":10391,"lbm_reads_lt_1ms":399,"lbm_write_time_us":18310,"lbm_writes_lt_1ms":380,"num_input_rowsets":2,"peak_mem_usage":4866694,"rows_written":305300,"spinlock_wait_cycles":384}
I20240503 08:12:42.232088 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3424387 bytes on disk
I20240503 08:12:42.232304 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:42.266958 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.393s user 0.280s sys 0.030s Metrics: {"bytes_written":8483281,"cfile_cache_hit":12,"cfile_cache_hit_bytes":857792,"cfile_cache_miss":376,"cfile_cache_miss_bytes":21405870,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":1017,"drs_written":1,"lbm_read_time_us":12971,"lbm_reads_lt_1ms":396,"lbm_write_time_us":17398,"lbm_writes_lt_1ms":377,"num_input_rowsets":2,"peak_mem_usage":4847634,"rows_written":304050,"spinlock_wait_cycles":896}
I20240503 08:12:42.267340 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:42.285144 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.016s user 0.013s sys 0.001s Metrics: {"bytes_written":290922,"cfile_init":1,"dirs.queue_time_us":263,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10150}
I20240503 08:12:42.285400 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3503251 bytes on disk
I20240503 08:12:42.285748 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":93,"lbm_reads_lt_1ms":8}
I20240503 08:12:42.285938 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011484
I20240503 08:12:42.294396 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:42.314491 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.020s user 0.010s sys 0.002s Metrics: {"bytes_written":277718,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":946,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":681,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:42.315107 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1056284 bytes of WAL
I20240503 08:12:42.315259 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:42.315336 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000010 (ops 5587-6200)
I20240503 08:12:42.315889 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:42.316037 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011475
I20240503 08:12:42.474810 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.401s user 0.286s sys 0.032s Metrics: {"bytes_written":8640113,"cfile_cache_hit":12,"cfile_cache_hit_bytes":749567,"cfile_cache_miss":382,"cfile_cache_miss_bytes":21937175,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":353,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":895,"drs_written":1,"lbm_read_time_us":12570,"lbm_reads_lt_1ms":406,"lbm_write_time_us":16563,"lbm_writes_lt_1ms":385,"num_input_rowsets":2,"peak_mem_usage":4940293,"rows_written":309700,"spinlock_wait_cycles":1536}
I20240503 08:12:42.475155 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:42.505419 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.030s user 0.015s sys 0.000s Metrics: {"bytes_written":304191,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":838,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":744,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10500}
I20240503 08:12:42.505795 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3572248 bytes on disk
I20240503 08:12:42.506067 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":57,"lbm_reads_lt_1ms":8}
I20240503 08:12:42.506274 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011332
I20240503 08:12:42.634152 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.347s user 0.282s sys 0.036s Metrics: {"bytes_written":8763120,"cfile_cache_hit":12,"cfile_cache_hit_bytes":520555,"cfile_cache_miss":386,"cfile_cache_miss_bytes":22516881,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":292,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":878,"drs_written":1,"lbm_read_time_us":10511,"lbm_reads_lt_1ms":406,"lbm_write_time_us":16773,"lbm_writes_lt_1ms":389,"num_input_rowsets":2,"peak_mem_usage":5022269,"rows_written":314200,"spinlock_wait_cycles":1536}
I20240503 08:12:42.634503 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3527841 bytes on disk
I20240503 08:12:42.634754 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:42.645049 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:42.668259 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":276460,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":829,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1639,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:12:42.668563 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1055658 bytes of WAL
I20240503 08:12:42.668738 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:42.669366 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000010 (ops 5674-6293)
I20240503 08:12:42.669826 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:42.670280 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 88296 bytes on disk
I20240503 08:12:42.670490 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:42.670673 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011293
I20240503 08:12:42.694833 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.379s user 0.299s sys 0.019s Metrics: {"bytes_written":8787148,"cfile_cache_hit":12,"cfile_cache_hit_bytes":775957,"cfile_cache_miss":390,"cfile_cache_miss_bytes":22316845,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":637,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":917,"drs_written":1,"lbm_read_time_us":12297,"lbm_reads_lt_1ms":414,"lbm_write_time_us":21926,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":388,"num_input_rowsets":2,"peak_mem_usage":5034953,"rows_written":314950,"spinlock_wait_cycles":1280}
I20240503 08:12:42.695019 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:42.711704 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":288120,"cfile_init":1,"dirs.queue_time_us":247,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1076,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10050}
I20240503 08:12:42.711879 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3628543 bytes on disk
I20240503 08:12:42.712023 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":35,"lbm_reads_lt_1ms":8}
I20240503 08:12:42.712132 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011292
I20240503 08:12:42.961829 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.455s user 0.299s sys 0.030s Metrics: {"bytes_written":8931373,"cfile_cache_hit":11,"cfile_cache_hit_bytes":496792,"cfile_cache_miss":395,"cfile_cache_miss_bytes":22977640,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":529,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":838,"drs_written":1,"lbm_read_time_us":15922,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":413,"lbm_write_time_us":19856,"lbm_writes_lt_1ms":397,"num_input_rowsets":2,"peak_mem_usage":5119416,"rows_written":320200,"spinlock_wait_cycles":768}
I20240503 08:12:42.962034 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3597794 bytes on disk
I20240503 08:12:42.962220 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:42.962646 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:42.989393 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.027s user 0.015s sys 0.003s Metrics: {"bytes_written":329429,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":857,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":744,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11450}
I20240503 08:12:42.989594 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1049945 bytes of WAL
I20240503 08:12:42.989755 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:42.989801 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000011 (ops 5840-6417)
I20240503 08:12:42.990322 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:42.990443 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011436
I20240503 08:12:43.068234 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.356s user 0.305s sys 0.029s Metrics: {"bytes_written":9064960,"cfile_cache_hit":12,"cfile_cache_hit_bytes":650665,"cfile_cache_miss":398,"cfile_cache_miss_bytes":23181827,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2520,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":688,"drs_written":1,"lbm_read_time_us":13453,"lbm_reads_lt_1ms":418,"lbm_write_time_us":19695,"lbm_writes_lt_1ms":402,"num_input_rowsets":2,"peak_mem_usage":5197859,"rows_written":325000,"spinlock_wait_cycles":3456}
I20240503 08:12:43.068418 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3652527 bytes on disk
I20240503 08:12:43.068549 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:43.078773 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:43.101496 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":275107,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":934,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2223,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:43.101788 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011404
I20240503 08:12:43.167045 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.496s user 0.321s sys 0.015s Metrics: {"bytes_written":9029044,"cfile_cache_hit":12,"cfile_cache_hit_bytes":740750,"cfile_cache_miss":398,"cfile_cache_miss_bytes":23003772,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":356,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":909,"drs_written":1,"lbm_read_time_us":16044,"lbm_reads_lt_1ms":418,"lbm_write_time_us":20859,"lbm_writes_lt_1ms":401,"num_input_rowsets":2,"peak_mem_usage":5178809,"rows_written":323800,"spinlock_wait_cycles":1156864,"thread_start_us":269,"threads_started":5}
I20240503 08:12:43.167292 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:43.213011 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.045s user 0.014s sys 0.004s Metrics: {"bytes_written":367804,"cfile_init":1,"dirs.queue_time_us":203,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":2151,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4657,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":12750}
I20240503 08:12:43.213307 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011392
I20240503 08:12:43.438513 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.448s user 0.314s sys 0.033s Metrics: {"bytes_written":9247361,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1135022,"cfile_cache_miss":405,"cfile_cache_miss_bytes":23190968,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1311,"drs_written":1,"lbm_read_time_us":12842,"lbm_reads_lt_1ms":429,"lbm_write_time_us":25034,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":410,"num_input_rowsets":2,"peak_mem_usage":5304514,"rows_written":331650,"spinlock_wait_cycles":4096}
I20240503 08:12:43.438815 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3730638 bytes on disk
I20240503 08:12:43.439333 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:43.439975 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:43.473765 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.033s user 0.020s sys 0.000s Metrics: {"bytes_written":354162,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":781,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12250}
I20240503 08:12:43.474046 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 113897 bytes on disk
I20240503 08:12:43.474192 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:43.474865 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011379
I20240503 08:12:43.589993 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.488s user 0.317s sys 0.025s Metrics: {"bytes_written":9326900,"cfile_cache_hit":10,"cfile_cache_hit_bytes":446022,"cfile_cache_miss":412,"cfile_cache_miss_bytes":24092260,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":843,"drs_written":1,"lbm_read_time_us":12378,"lbm_reads_lt_1ms":436,"lbm_write_time_us":28268,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":412,"num_input_rowsets":2,"peak_mem_usage":5353534,"rows_written":334550,"spinlock_wait_cycles":3328,"thread_start_us":217,"threads_started":5}
I20240503 08:12:43.590310 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:43.630138 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.040s user 0.016s sys 0.004s Metrics: {"bytes_written":396903,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":2697,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1265,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":13700}
I20240503 08:12:43.630468 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1055209 bytes of WAL
I20240503 08:12:43.630625 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:43.630694 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000011 (ops 6201-6796)
I20240503 08:12:43.631232 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:43.631404 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3892073 bytes on disk
I20240503 08:12:43.631595 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":48,"lbm_reads_lt_1ms":8}
I20240503 08:12:43.631778 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011333
I20240503 08:12:43.791783 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.578s user 0.324s sys 0.029s Metrics: {"bytes_written":9380800,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1036255,"cfile_cache_miss":410,"cfile_cache_miss_bytes":23642455,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":430,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1046,"drs_written":1,"lbm_read_time_us":28501,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":434,"lbm_write_time_us":26811,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":415,"num_input_rowsets":2,"peak_mem_usage":5382832,"rows_written":336550,"spinlock_wait_cycles":3968,"thread_start_us":269,"threads_started":5}
I20240503 08:12:43.792035 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:43.837399 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.045s user 0.022s sys 0.003s Metrics: {"bytes_written":521430,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5216,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":18300}
I20240503 08:12:43.837862 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1056833 bytes of WAL
I20240503 08:12:43.837985 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:43.838034 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000011 (ops 6294-6898)
I20240503 08:12:43.839488 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:43.839648 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3961001 bytes on disk
I20240503 08:12:43.840750 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":439,"lbm_reads_lt_1ms":8}
I20240503 08:12:43.840921 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011246
I20240503 08:12:43.924041 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.449s user 0.333s sys 0.013s Metrics: {"bytes_written":9589092,"cfile_cache_hit":13,"cfile_cache_hit_bytes":861703,"cfile_cache_miss":423,"cfile_cache_miss_bytes":24364607,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":1407,"dirs.run_wall_time_us":28403,"drs_written":1,"lbm_read_time_us":14976,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":442,"lbm_write_time_us":23026,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":424,"num_input_rowsets":2,"peak_mem_usage":5502763,"rows_written":343900,"spinlock_wait_cycles":5888}
I20240503 08:12:43.927153 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:43.977193 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.050s user 0.023s sys 0.002s Metrics: {"bytes_written":475456,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1073,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1601,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16650}
I20240503 08:12:43.977447 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1058860 bytes of WAL
I20240503 08:12:43.977818 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:43.977890 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000012 (ops 6418-7000)
I20240503 08:12:43.979507 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:43.979835 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011174
I20240503 08:12:44.002130 17235 test_util.cc:268] Using random seed: 1220747086
I20240503 08:12:44.011179 17235 test_workload.cc:401] TestWorkload: Skipping table creation because table test-workload already exists
I20240503 08:12:44.126504 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.495s user 0.311s sys 0.035s Metrics: {"bytes_written":9706292,"cfile_cache_hit":11,"cfile_cache_hit_bytes":568020,"cfile_cache_miss":431,"cfile_cache_miss_bytes":24973226,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":839,"drs_written":1,"lbm_read_time_us":15512,"lbm_reads_lt_1ms":451,"lbm_write_time_us":28428,"lbm_writes_1-10_ms":6,"lbm_writes_lt_1ms":425,"num_input_rowsets":2,"peak_mem_usage":5571990,"rows_written":348250,"spinlock_wait_cycles":3456,"thread_start_us":205,"threads_started":5}
I20240503 08:12:44.126878 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 3923645 bytes on disk
I20240503 08:12:44.127552 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:44.127719 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:44.174661 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.047s user 0.027s sys 0.001s Metrics: {"bytes_written":502078,"cfile_init":1,"drs_written":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2575,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":17550}
I20240503 08:12:44.174938 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011109
I20240503 08:12:44.288910 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.447s user 0.349s sys 0.009s Metrics: {"bytes_written":9889489,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1072419,"cfile_cache_miss":436,"cfile_cache_miss_bytes":24938837,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":2181,"drs_written":1,"lbm_read_time_us":14300,"lbm_reads_lt_1ms":456,"lbm_write_time_us":20530,"lbm_writes_lt_1ms":438,"num_input_rowsets":2,"peak_mem_usage":5674630,"rows_written":354850,"spinlock_wait_cycles":2048}
I20240503 08:12:44.289256 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:44.314906 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.018s sys 0.003s Metrics: {"bytes_written":456323,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":693,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1646,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15900}
I20240503 08:12:44.315209 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011053
I20240503 08:12:44.467391 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.487s user 0.327s sys 0.039s Metrics: {"bytes_written":10046823,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1229448,"cfile_cache_miss":441,"cfile_cache_miss_bytes":25216684,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":982,"drs_written":1,"lbm_read_time_us":17993,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":466,"lbm_write_time_us":24890,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":443,"num_input_rowsets":2,"peak_mem_usage":5768903,"rows_written":360550,"spinlock_wait_cycles":4224}
I20240503 08:12:44.474519 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4065919 bytes on disk
I20240503 08:12:44.478806 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:44.479007 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:44.553754 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.075s user 0.022s sys 0.000s Metrics: {"bytes_written":422241,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":903,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4258,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":14700}
I20240503 08:12:44.554045 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 137919 bytes on disk
I20240503 08:12:44.554286 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240503 08:12:44.554531 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.011005
I20240503 08:12:44.665256 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.490s user 0.363s sys 0.013s Metrics: {"bytes_written":10189148,"cfile_cache_hit":13,"cfile_cache_hit_bytes":683693,"cfile_cache_miss":451,"cfile_cache_miss_bytes":26145481,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":212,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":878,"drs_written":1,"lbm_read_time_us":15005,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":474,"lbm_write_time_us":41307,"lbm_writes_1-10_ms":6,"lbm_writes_lt_1ms":445,"num_input_rowsets":2,"peak_mem_usage":5853304,"rows_written":365800,"spinlock_wait_cycles":1664,"thread_start_us":79,"threads_started":2}
I20240503 08:12:44.666043 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4126873 bytes on disk
I20240503 08:12:44.666869 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:44.667897 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:44.690443 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.022s user 0.017s sys 0.002s Metrics: {"bytes_written":429834,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":1791,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1245,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":15000}
I20240503 08:12:44.690730 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1049624 bytes of WAL
I20240503 08:12:44.690850 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:44.690920 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000012 (ops 6797-7410)
I20240503 08:12:44.691351 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:44.691473 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 140591 bytes on disk
I20240503 08:12:44.691620 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:44.691748 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010951
I20240503 08:12:44.710188 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.395s user 0.350s sys 0.022s Metrics: {"bytes_written":10328840,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1063293,"cfile_cache_miss":454,"cfile_cache_miss_bytes":26130685,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":1127,"drs_written":1,"lbm_read_time_us":11793,"lbm_reads_lt_1ms":482,"lbm_write_time_us":19108,"lbm_writes_lt_1ms":456,"num_input_rowsets":2,"peak_mem_usage":5935388,"rows_written":370750,"spinlock_wait_cycles":23552}
I20240503 08:12:44.710736 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:44.727950 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.017s user 0.011s sys 0.004s Metrics: {"bytes_written":322751,"cfile_init":1,"dirs.queue_time_us":31,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":758,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11200}
I20240503 08:12:44.728217 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1089510 bytes of WAL
I20240503 08:12:44.728344 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:44.728406 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000012 (ops 6899-7555)
I20240503 08:12:44.728827 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:44.728940 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4288440 bytes on disk
I20240503 08:12:44.729166 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":34,"lbm_reads_lt_1ms":8}
I20240503 08:12:44.729312 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010875
I20240503 08:12:45.097744 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.406s user 0.375s sys 0.009s Metrics: {"bytes_written":10605100,"cfile_cache_hit":10,"cfile_cache_hit_bytes":529043,"cfile_cache_miss":470,"cfile_cache_miss_bytes":27405063,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":221,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":971,"drs_written":1,"lbm_read_time_us":13045,"lbm_reads_lt_1ms":490,"lbm_write_time_us":18709,"lbm_writes_lt_1ms":468,"num_input_rowsets":2,"peak_mem_usage":6099594,"rows_written":380800,"spinlock_wait_cycles":2432}
I20240503 08:12:45.098070 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:45.118927 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.390s user 0.356s sys 0.028s Metrics: {"bytes_written":10638774,"cfile_cache_hit":10,"cfile_cache_hit_bytes":478392,"cfile_cache_miss":470,"cfile_cache_miss_bytes":27554664,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":914,"drs_written":1,"lbm_read_time_us":12934,"lbm_reads_lt_1ms":490,"lbm_write_time_us":17546,"lbm_writes_lt_1ms":470,"num_input_rowsets":2,"peak_mem_usage":6122079,"rows_written":381950,"spinlock_wait_cycles":4096}
I20240503 08:12:45.119287 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4314061 bytes on disk
I20240503 08:12:45.119489 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:45.119643 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:45.144613 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.021s sys 0.000s Metrics: {"bytes_written":349083,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1052,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12050}
I20240503 08:12:45.145063 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010845
I20240503 08:12:45.149353 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.051s user 0.018s sys 0.003s Metrics: {"bytes_written":358443,"cfile_init":1,"dirs.queue_time_us":28,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3420,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":12400}
I20240503 08:12:45.149587 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4418586 bytes on disk
I20240503 08:12:45.149811 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20240503 08:12:45.150004 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010854
I20240503 08:12:45.402586 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.848s user 0.336s sys 0.050s Metrics: {"bytes_written":10453750,"cfile_cache_hit":13,"cfile_cache_hit_bytes":898023,"cfile_cache_miss":461,"cfile_cache_miss_bytes":26633699,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":12053,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":805,"drs_written":1,"lbm_read_time_us":27561,"lbm_reads_1-10_ms":6,"lbm_reads_lt_1ms":475,"lbm_write_time_us":41946,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":459,"num_input_rowsets":2,"peak_mem_usage":6009712,"rows_written":375250,"spinlock_wait_cycles":3220736,"thread_start_us":344,"threads_started":6}
I20240503 08:12:45.402976 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:45.479727 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.077s user 0.023s sys 0.012s Metrics: {"bytes_written":765819,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":740,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3320,"lbm_writes_lt_1ms":51,"peak_mem_usage":0,"rows_written":26900}
I20240503 08:12:45.481123 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1054869 bytes of WAL
I20240503 08:12:45.481314 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:45.481405 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000013 (ops 7001-7581)
I20240503 08:12:45.481935 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:45.482084 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4501156 bytes on disk
I20240503 08:12:45.482400 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":137,"lbm_reads_lt_1ms":8}
I20240503 08:12:45.482681 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010764
I20240503 08:12:45.548416 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.403s user 0.375s sys 0.024s Metrics: {"bytes_written":10970818,"cfile_cache_hit":14,"cfile_cache_hit_bytes":987259,"cfile_cache_miss":480,"cfile_cache_miss_bytes":27931767,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":467,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":1171,"drs_written":1,"lbm_read_time_us":11745,"lbm_reads_lt_1ms":504,"lbm_write_time_us":19116,"lbm_writes_lt_1ms":486,"num_input_rowsets":2,"peak_mem_usage":6317028,"rows_written":394000,"spinlock_wait_cycles":4480}
I20240503 08:12:45.548774 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4454838 bytes on disk
I20240503 08:12:45.548955 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:45.549098 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:45.570735 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.022s user 0.018s sys 0.000s Metrics: {"bytes_written":347586,"cfile_init":1,"dirs.queue_time_us":29,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":838,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12000}
I20240503 08:12:45.571003 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 111466 bytes on disk
I20240503 08:12:45.571182 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:45.571363 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010391
I20240503 08:12:45.590163 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.440s user 0.352s sys 0.045s Metrics: {"bytes_written":10946063,"cfile_cache_hit":12,"cfile_cache_hit_bytes":670211,"cfile_cache_miss":480,"cfile_cache_miss_bytes":28189369,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":794,"drs_written":1,"lbm_read_time_us":12335,"lbm_reads_lt_1ms":500,"lbm_write_time_us":20417,"lbm_writes_lt_1ms":482,"num_input_rowsets":2,"peak_mem_usage":6303679,"rows_written":393200,"spinlock_wait_cycles":3968}
I20240503 08:12:45.590483 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4444989 bytes on disk
I20240503 08:12:45.590737 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20240503 08:12:45.590884 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:45.616439 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.018s sys 0.001s Metrics: {"bytes_written":400800,"cfile_init":1,"dirs.queue_time_us":26,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1964,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":13850}
I20240503 08:12:45.616632 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1051131 bytes of WAL
I20240503 08:12:45.622826 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:45.623062 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000013 (ops 7411-8059)
I20240503 08:12:45.623940 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:45.624159 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010382
I20240503 08:12:45.972172 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.401s user 0.356s sys 0.039s Metrics: {"bytes_written":11302388,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1174565,"cfile_cache_miss":497,"cfile_cache_miss_bytes":28624239,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":886,"drs_written":1,"lbm_read_time_us":14179,"lbm_reads_lt_1ms":517,"lbm_write_time_us":20128,"lbm_writes_lt_1ms":500,"num_input_rowsets":2,"peak_mem_usage":6512062,"rows_written":406000,"spinlock_wait_cycles":2688}
I20240503 08:12:45.972425 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:45.993929 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.021s user 0.019s sys 0.000s Metrics: {"bytes_written":408298,"cfile_init":1,"dirs.queue_time_us":30,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":900,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14150}
I20240503 08:12:45.994194 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1052039 bytes of WAL
I20240503 08:12:45.994318 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:45.994390 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000013 (ops 7556-8160)
I20240503 08:12:45.994971 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:45.995108 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4727749 bytes on disk
I20240503 08:12:45.995294 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20240503 08:12:45.995432 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010258
I20240503 08:12:46.181143 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.557s user 0.376s sys 0.038s Metrics: {"bytes_written":11332973,"cfile_cache_hit":12,"cfile_cache_hit_bytes":732147,"cfile_cache_miss":498,"cfile_cache_miss_bytes":29140099,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":803,"drs_written":1,"lbm_read_time_us":17158,"lbm_reads_lt_1ms":522,"lbm_write_time_us":32573,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":497,"num_input_rowsets":2,"peak_mem_usage":6527276,"rows_written":407050,"spinlock_wait_cycles":12416,"thread_start_us":262,"threads_started":5}
I20240503 08:12:46.181499 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:46.217528 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.036s user 0.022s sys 0.004s Metrics: {"bytes_written":534441,"cfile_init":1,"dirs.queue_time_us":24,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3745,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":18800}
I20240503 08:12:46.217800 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010204
I20240503 08:12:46.268447 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.786s user 0.381s sys 0.036s Metrics: {"bytes_written":11198452,"cfile_cache_hit":13,"cfile_cache_hit_bytes":842786,"cfile_cache_miss":489,"cfile_cache_miss_bytes":28652780,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":302,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":2476,"drs_written":1,"lbm_read_time_us":35218,"lbm_reads_1-10_ms":7,"lbm_reads_lt_1ms":502,"lbm_write_time_us":34041,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":493,"num_input_rowsets":2,"peak_mem_usage":6434593,"rows_written":402150,"spinlock_wait_cycles":1152,"thread_start_us":339,"threads_started":6}
I20240503 08:12:46.269076 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4549738 bytes on disk
I20240503 08:12:46.269362 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 08:12:46.269680 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:46.310978 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.040s user 0.032s sys 0.000s Metrics: {"bytes_written":754253,"cfile_init":1,"dirs.queue_time_us":1090,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":3234,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1544,"lbm_writes_lt_1ms":51,"peak_mem_usage":0,"rows_written":26450}
I20240503 08:12:46.311291 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1051797 bytes of WAL
I20240503 08:12:46.311468 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:46.311545 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000014 (ops 7582-8160)
I20240503 08:12:46.312146 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:46.312302 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 260289 bytes on disk
I20240503 08:12:46.312505 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:46.312659 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010421
I20240503 08:12:46.441810 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.446s user 0.390s sys 0.031s Metrics: {"bytes_written":11691470,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1053458,"cfile_cache_miss":516,"cfile_cache_miss_bytes":29786880,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":209,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1237,"drs_written":1,"lbm_read_time_us":14070,"lbm_reads_lt_1ms":536,"lbm_write_time_us":22346,"lbm_writes_lt_1ms":516,"num_input_rowsets":2,"peak_mem_usage":6739834,"rows_written":420150,"spinlock_wait_cycles":5120}
I20240503 08:12:46.442276 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:46.466063 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.024s user 0.016s sys 0.003s Metrics: {"bytes_written":367784,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":4029,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12750}
I20240503 08:12:46.466415 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4876542 bytes on disk
I20240503 08:12:46.466877 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":42,"lbm_reads_lt_1ms":8}
I20240503 08:12:46.467118 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010259
I20240503 08:12:46.710427 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.492s user 0.391s sys 0.048s Metrics: {"bytes_written":11854213,"cfile_cache_hit":12,"cfile_cache_hit_bytes":746011,"cfile_cache_miss":522,"cfile_cache_miss_bytes":30502551,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":16320,"lbm_reads_lt_1ms":550,"lbm_write_time_us":27330,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":521,"num_input_rowsets":2,"peak_mem_usage":6827800,"rows_written":425850,"spinlock_wait_cycles":5120}
I20240503 08:12:46.711004 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4824267 bytes on disk
I20240503 08:12:46.712556 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:46.712812 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:46.740386 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.027s user 0.015s sys 0.004s Metrics: {"bytes_written":373198,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":5029,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":798,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12950,"spinlock_wait_cycles":2304}
I20240503 08:12:46.741976 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1048348 bytes of WAL
I20240503 08:12:46.742130 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:46.742204 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000014 (ops 8060-8707)
I20240503 08:12:46.742703 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:46.742985 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 120617 bytes on disk
I20240503 08:12:46.743186 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:46.743357 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010226
I20240503 08:12:46.900125 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.587s user 0.393s sys 0.043s Metrics: {"bytes_written":11928856,"cfile_cache_hit":12,"cfile_cache_hit_bytes":767401,"cfile_cache_miss":526,"cfile_cache_miss_bytes":30680343,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":102,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":890,"drs_written":1,"lbm_read_time_us":24693,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":545,"lbm_write_time_us":28852,"lbm_writes_lt_1ms":525,"num_input_rowsets":2,"peak_mem_usage":6864824,"rows_written":428600,"spinlock_wait_cycles":6400,"thread_start_us":318,"threads_started":5}
I20240503 08:12:46.900466 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4856869 bytes on disk
I20240503 08:12:46.900617 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20240503 08:12:46.900810 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:46.928682 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.028s user 0.019s sys 0.002s Metrics: {"bytes_written":462558,"cfile_init":1,"dirs.queue_time_us":273,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16150}
I20240503 08:12:46.928977 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1115031 bytes of WAL
I20240503 08:12:46.929175 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:46.929273 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000015 (ops 8161-8774)
I20240503 08:12:46.929859 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:46.930016 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010315
I20240503 08:12:46.977969 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.511s user 0.392s sys 0.037s Metrics: {"bytes_written":12046664,"cfile_cache_hit":11,"cfile_cache_hit_bytes":668657,"cfile_cache_miss":531,"cfile_cache_miss_bytes":31108767,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":1447,"drs_written":1,"lbm_read_time_us":17227,"lbm_reads_lt_1ms":551,"lbm_write_time_us":21762,"lbm_writes_lt_1ms":531,"num_input_rowsets":2,"peak_mem_usage":6948414,"rows_written":432900,"spinlock_wait_cycles":2169216,"thread_start_us":311,"threads_started":5}
I20240503 08:12:46.979004 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 4906637 bytes on disk
I20240503 08:12:46.979154 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20240503 08:12:46.979324 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:47.025655 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.046s user 0.020s sys 0.000s Metrics: {"bytes_written":414678,"cfile_init":1,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":875,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1553,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14400}
I20240503 08:12:47.025955 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1052862 bytes of WAL
I20240503 08:12:47.027150 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:47.027324 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000014 (ops 8161-8758)
I20240503 08:12:47.027824 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:47.027995 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 135728 bytes on disk
I20240503 08:12:47.028340 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:47.028535 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010287
I20240503 08:12:47.387804 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.644s user 0.402s sys 0.055s Metrics: {"bytes_written":12209727,"cfile_cache_hit":13,"cfile_cache_hit_bytes":872694,"cfile_cache_miss":533,"cfile_cache_miss_bytes":31344750,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":14011,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1560,"drs_written":1,"lbm_read_time_us":23971,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":550,"lbm_write_time_us":35393,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":534,"num_input_rowsets":2,"peak_mem_usage":7044082,"rows_written":438800,"spinlock_wait_cycles":24320,"thread_start_us":310,"threads_started":6}
I20240503 08:12:47.388586 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:47.432241 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.043s user 0.025s sys 0.004s Metrics: {"bytes_written":606567,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":824,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1386,"lbm_writes_lt_1ms":45,"peak_mem_usage":0,"rows_written":21250}
I20240503 08:12:47.432562 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010137
I20240503 08:12:47.503652 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.573s user 0.389s sys 0.054s Metrics: {"bytes_written":12374824,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1217640,"cfile_cache_miss":541,"cfile_cache_miss_bytes":31431830,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":841,"drs_written":1,"lbm_read_time_us":20880,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":564,"lbm_write_time_us":30360,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":543,"num_input_rowsets":2,"peak_mem_usage":7137556,"rows_written":444750,"spinlock_wait_cycles":4992,"thread_start_us":222,"threads_started":5}
I20240503 08:12:47.504036 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5043647 bytes on disk
I20240503 08:12:47.504520 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:47.504674 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:47.543972 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.039s user 0.023s sys 0.002s Metrics: {"bytes_written":531825,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":3844,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1409,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18700}
I20240503 08:12:47.544323 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 178374 bytes on disk
I20240503 08:12:47.544574 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20240503 08:12:47.545083 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.010081
I20240503 08:12:47.735703 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.707s user 0.393s sys 0.062s Metrics: {"bytes_written":12443491,"cfile_cache_hit":13,"cfile_cache_hit_bytes":893418,"cfile_cache_miss":547,"cfile_cache_miss_bytes":31945662,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":12332,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":25922,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":564,"lbm_write_time_us":33271,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":545,"num_input_rowsets":2,"peak_mem_usage":7180746,"rows_written":447300,"spinlock_wait_cycles":19584,"thread_start_us":343,"threads_started":6}
I20240503 08:12:47.736140 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5073596 bytes on disk
I20240503 08:12:47.736364 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:47.736521 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:47.798900 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.062s user 0.030s sys 0.001s Metrics: {"bytes_written":690314,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":809,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2887,"lbm_writes_lt_1ms":49,"peak_mem_usage":0,"rows_written":24150}
I20240503 08:12:47.799440 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1063018 bytes of WAL
I20240503 08:12:47.799616 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:47.799696 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000015 (ops 8759-9363)
I20240503 08:12:47.800315 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:47.800967 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 232405 bytes on disk
I20240503 08:12:47.801173 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:47.801366 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009998
I20240503 08:12:48.113839 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.569s user 0.426s sys 0.037s Metrics: {"bytes_written":12893444,"cfile_cache_hit":12,"cfile_cache_hit_bytes":848228,"cfile_cache_miss":566,"cfile_cache_miss_bytes":33166608,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1277,"drs_written":1,"lbm_read_time_us":20451,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":585,"lbm_write_time_us":33687,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":566,"num_input_rowsets":2,"peak_mem_usage":7438861,"rows_written":463450,"spinlock_wait_cycles":6528,"thread_start_us":218,"threads_started":5}
I20240503 08:12:48.114217 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5261414 bytes on disk
I20240503 08:12:48.114408 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:48.115293 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:48.149868 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.034s user 0.028s sys 0.004s Metrics: {"bytes_written":628634,"cfile_init":1,"dirs.queue_time_us":564,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1734,"lbm_writes_lt_1ms":46,"peak_mem_usage":0,"rows_written":22100}
I20240503 08:12:48.150177 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1096961 bytes of WAL
I20240503 08:12:48.150283 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:48.150315 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000016 (ops 8775-9378)
I20240503 08:12:48.151175 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:48.151497 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009542
I20240503 08:12:48.263201 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.830s user 0.429s sys 0.049s Metrics: {"bytes_written":12797332,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1612380,"cfile_cache_miss":554,"cfile_cache_miss_bytes":32155538,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":13032,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":1994,"drs_written":1,"lbm_read_time_us":28405,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":578,"lbm_write_time_us":45141,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":559,"num_input_rowsets":2,"peak_mem_usage":7381062,"rows_written":460050,"spinlock_wait_cycles":4944128,"thread_start_us":339,"threads_started":6}
I20240503 08:12:48.263545 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:48.332284 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.068s user 0.035s sys 0.001s Metrics: {"bytes_written":830216,"cfile_init":1,"dirs.queue_time_us":562,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":941,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2782,"lbm_writes_lt_1ms":55,"peak_mem_usage":0,"rows_written":29100}
I20240503 08:12:48.332569 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1054354 bytes of WAL
I20240503 08:12:48.333040 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:48.333122 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000015 (ops 8708-9341)
I20240503 08:12:48.333554 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:48.333679 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5508310 bytes on disk
I20240503 08:12:48.334671 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":106,"lbm_reads_lt_1ms":8}
I20240503 08:12:48.334831 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009728
I20240503 08:12:48.652565 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.851s user 0.416s sys 0.069s Metrics: {"bytes_written":13111464,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1342357,"cfile_cache_miss":570,"cfile_cache_miss_bytes":33253581,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":11715,"dirs.run_cpu_time_us":1116,"dirs.run_wall_time_us":6712,"drs_written":1,"lbm_read_time_us":26315,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":588,"lbm_write_time_us":44337,"lbm_writes_1-10_ms":6,"lbm_writes_lt_1ms":571,"num_input_rowsets":2,"peak_mem_usage":7563394,"rows_written":471450,"spinlock_wait_cycles":81920,"thread_start_us":239,"threads_started":5}
I20240503 08:12:48.654919 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:48.691818 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.540s user 0.453s sys 0.036s Metrics: {"bytes_written":13500118,"cfile_cache_hit":12,"cfile_cache_hit_bytes":755043,"cfile_cache_miss":594,"cfile_cache_miss_bytes":34889985,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":472,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":928,"drs_written":1,"lbm_read_time_us":17540,"lbm_reads_lt_1ms":618,"lbm_write_time_us":29604,"lbm_writes_lt_1ms":593,"num_input_rowsets":2,"peak_mem_usage":7794553,"rows_written":485550,"spinlock_wait_cycles":274944}
I20240503 08:12:48.693039 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:48.728497 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.035s user 0.017s sys 0.006s Metrics: {"bytes_written":474046,"cfile_init":1,"dirs.queue_time_us":29,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1104,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16600}
I20240503 08:12:48.728726 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1056928 bytes of WAL
I20240503 08:12:48.729146 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:48.729192 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000017 (ops 9379-9960)
I20240503 08:12:48.732800 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":31104}
I20240503 08:12:48.733350 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009514
I20240503 08:12:48.742743 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.087s user 0.034s sys 0.004s Metrics: {"bytes_written":840196,"cfile_init":1,"dirs.queue_time_us":21,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5903,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":53,"peak_mem_usage":0,"rows_written":29500}
I20240503 08:12:48.743021 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1056255 bytes of WAL
I20240503 08:12:48.743157 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:48.743196 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000016 (ops 9364-9958)
I20240503 08:12:48.743825 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20240503 08:12:48.743957 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009533
I20240503 08:12:49.077142 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.742s user 0.459s sys 0.049s Metrics: {"bytes_written":13601968,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1409676,"cfile_cache_miss":594,"cfile_cache_miss_bytes":34487726,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":11749,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":4413,"drs_written":1,"lbm_read_time_us":21549,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":612,"lbm_write_time_us":39966,"lbm_writes_1-10_ms":7,"lbm_writes_lt_1ms":591,"num_input_rowsets":2,"peak_mem_usage":7846044,"rows_written":489150,"spinlock_wait_cycles":2356096,"thread_start_us":306,"threads_started":6}
I20240503 08:12:49.077515 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5559211 bytes on disk
I20240503 08:12:49.077673 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20240503 08:12:49.077873 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:49.123343 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.045s user 0.023s sys 0.008s Metrics: {"bytes_written":736021,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":937,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1508,"lbm_writes_lt_1ms":50,"peak_mem_usage":0,"rows_written":25750}
I20240503 08:12:49.123651 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1066350 bytes of WAL
I20240503 08:12:49.123833 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:49.123919 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000016 (ops 9342-9981)
I20240503 08:12:49.124368 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:49.124538 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 248786 bytes on disk
I20240503 08:12:49.124730 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:49.124907 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009506
I20240503 08:12:49.582123 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.848s user 0.472s sys 0.047s Metrics: {"bytes_written":13957526,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1580664,"cfile_cache_miss":608,"cfile_cache_miss_bytes":35292922,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":3942,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":9443,"drs_written":1,"lbm_read_time_us":40104,"lbm_reads_1-10_ms":6,"lbm_reads_lt_1ms":630,"lbm_write_time_us":50241,"lbm_writes_1-10_ms":9,"lbm_writes_lt_1ms":603,"num_input_rowsets":2,"peak_mem_usage":8069793,"rows_written":502150,"spinlock_wait_cycles":1174272,"thread_start_us":637,"threads_started":1}
I20240503 08:12:49.582495 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:49.584028 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.840s user 0.473s sys 0.046s Metrics: {"bytes_written":13924351,"cfile_cache_hit":19,"cfile_cache_hit_bytes":1885128,"cfile_cache_miss":605,"cfile_cache_miss_bytes":34885388,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":62540,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":35978,"lbm_reads_1-10_ms":6,"lbm_reads_lt_1ms":627,"lbm_write_time_us":44329,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":606,"num_input_rowsets":2,"peak_mem_usage":8037378,"rows_written":500950,"spinlock_wait_cycles":660608,"thread_start_us":340,"threads_started":6}
I20240503 08:12:49.584339 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:49.647143 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.062s user 0.036s sys 0.001s Metrics: {"bytes_written":830050,"cfile_init":1,"dirs.queue_time_us":22,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2092,"lbm_writes_lt_1ms":55,"peak_mem_usage":0,"rows_written":29100}
I20240503 08:12:49.647557 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1054877 bytes of WAL
I20240503 08:12:49.647742 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:49.647804 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000017 (ops 9959-10560)
I20240503 08:12:49.649950 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:49.650178 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5985130 bytes on disk
I20240503 08:12:49.650415 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20240503 08:12:49.650591 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009347
I20240503 08:12:49.662247 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.080s user 0.031s sys 0.008s Metrics: {"bytes_written":795812,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":8377,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2953,"lbm_writes_lt_1ms":53,"peak_mem_usage":0,"rows_written":27900}
I20240503 08:12:49.663215 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1053437 bytes of WAL
I20240503 08:12:49.663338 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:49.663453 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000018 (ops 9961-10540)
I20240503 08:12:49.663933 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:49.664098 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009347
I20240503 08:12:49.685035 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.560s user 0.464s sys 0.063s Metrics: {"bytes_written":14313582,"cfile_cache_hit":13,"cfile_cache_hit_bytes":922332,"cfile_cache_miss":625,"cfile_cache_miss_bytes":36871734,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":1204,"drs_written":1,"lbm_read_time_us":19192,"lbm_reads_lt_1ms":645,"lbm_write_time_us":27753,"lbm_writes_lt_1ms":628,"num_input_rowsets":2,"peak_mem_usage":8267808,"rows_written":514900,"spinlock_wait_cycles":449664,"thread_start_us":231,"threads_started":5}
I20240503 08:12:49.685384 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 5858317 bytes on disk
I20240503 08:12:49.685717 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20240503 08:12:49.686043 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:49.720876 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.035s user 0.024s sys 0.000s Metrics: {"bytes_written":501982,"cfile_init":1,"dirs.queue_time_us":44,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1282,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17550}
I20240503 08:12:49.721408 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 166259 bytes on disk
I20240503 08:12:49.721561 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:49.721735 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009097
I20240503 08:12:50.301954 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.580s user 0.498s sys 0.046s Metrics: {"bytes_written":14797689,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1529868,"cfile_cache_miss":645,"cfile_cache_miss_bytes":37570114,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":928,"drs_written":1,"lbm_read_time_us":21193,"lbm_reads_lt_1ms":665,"lbm_write_time_us":29580,"lbm_writes_lt_1ms":649,"num_input_rowsets":2,"peak_mem_usage":8561176,"rows_written":532450,"spinlock_wait_cycles":6056704,"thread_start_us":248,"threads_started":5}
I20240503 08:12:50.302343 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6062531 bytes on disk
I20240503 08:12:50.302953 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:50.303171 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:50.337456 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.034s user 0.023s sys 0.000s Metrics: {"bytes_written":437343,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1140,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":15300}
I20240503 08:12:50.337697 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1123690 bytes of WAL
I20240503 08:12:50.337769 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:50.337790 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000017 (ops 9982-10663)
I20240503 08:12:50.338286 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:50.338497 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009113
I20240503 08:12:50.415243 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.765s user 0.479s sys 0.071s Metrics: {"bytes_written":14728104,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1247483,"cfile_cache_miss":643,"cfile_cache_miss_bytes":37667777,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":14704,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":3041,"drs_written":1,"lbm_read_time_us":24296,"lbm_reads_lt_1ms":663,"lbm_write_time_us":36780,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":644,"num_input_rowsets":2,"peak_mem_usage":8510623,"rows_written":530050,"spinlock_wait_cycles":5939072,"thread_start_us":1779,"threads_started":6}
I20240503 08:12:50.415596 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6034275 bytes on disk
I20240503 08:12:50.415807 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:50.415925 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:50.462667 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.047s user 0.029s sys 0.000s Metrics: {"bytes_written":602382,"cfile_init":1,"dirs.queue_time_us":549,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":1413,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":7777,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":21100}
I20240503 08:12:50.462976 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 204820 bytes on disk
I20240503 08:12:50.463210 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:50.464890 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.801s user 0.498s sys 0.061s Metrics: {"bytes_written":14728104,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1559424,"cfile_cache_miss":639,"cfile_cache_miss_bytes":37354146,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":11558,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":2177,"drs_written":1,"lbm_read_time_us":28763,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":664,"lbm_write_time_us":39400,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":643,"num_input_rowsets":2,"peak_mem_usage":8511830,"rows_written":530050,"spinlock_wait_cycles":4306944,"thread_start_us":413,"threads_started":6}
I20240503 08:12:50.464998 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009146
I20240503 08:12:50.465140 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6034275 bytes on disk
I20240503 08:12:50.468523 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1377,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 08:12:50.468761 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:50.527244 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.057s user 0.027s sys 0.004s Metrics: {"bytes_written":652174,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":12266,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1567,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":22850}
I20240503 08:12:50.527549 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.009128
I20240503 08:12:50.951768 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.613s user 0.519s sys 0.032s Metrics: {"bytes_written":15219917,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1208349,"cfile_cache_miss":667,"cfile_cache_miss_bytes":39032467,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":20265,"lbm_reads_lt_1ms":691,"lbm_write_time_us":30369,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":666,"num_input_rowsets":2,"peak_mem_usage":8812040,"rows_written":547750,"spinlock_wait_cycles":2798464,"thread_start_us":205,"threads_started":5}
I20240503 08:12:50.952152 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:50.992024 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.040s user 0.027s sys 0.000s Metrics: {"bytes_written":596047,"cfile_init":1,"dirs.queue_time_us":27,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1422,"lbm_writes_lt_1ms":45,"peak_mem_usage":0,"rows_written":20850}
I20240503 08:12:50.992384 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1058117 bytes of WAL
I20240503 08:12:50.992487 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:50.992558 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000018 (ops 10664-11318)
I20240503 08:12:50.993049 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:50.993398 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6439608 bytes on disk
I20240503 08:12:50.993610 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":34,"lbm_reads_lt_1ms":8}
I20240503 08:12:50.993777 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008992
I20240503 08:12:51.283985 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.819s user 0.500s sys 0.061s Metrics: {"bytes_written":15316317,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1538241,"cfile_cache_miss":667,"cfile_cache_miss_bytes":38938225,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":11029,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":8010,"drs_written":1,"lbm_read_time_us":32241,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":683,"lbm_write_time_us":37622,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":666,"num_input_rowsets":2,"peak_mem_usage":8861482,"rows_written":551150,"spinlock_wait_cycles":1806976,"thread_start_us":846,"threads_started":6}
I20240503 08:12:51.284353 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6280131 bytes on disk
I20240503 08:12:51.284502 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20240503 08:12:51.284758 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:51.336907 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.809s user 0.529s sys 0.034s Metrics: {"bytes_written":15365097,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1542306,"cfile_cache_miss":669,"cfile_cache_miss_bytes":39061848,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":20333,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":985,"drs_written":1,"lbm_read_time_us":25182,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":691,"lbm_write_time_us":42766,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":667,"num_input_rowsets":2,"peak_mem_usage":8888165,"rows_written":552900,"spinlock_wait_cycles":2844032,"thread_start_us":422,"threads_started":6}
I20240503 08:12:51.337216 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6299784 bytes on disk
I20240503 08:12:51.338200 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:51.338596 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:51.346104 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.061s user 0.035s sys 0.002s Metrics: {"bytes_written":851270,"cfile_init":1,"dirs.queue_time_us":99,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":1133,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3594,"lbm_writes_lt_1ms":55,"peak_mem_usage":0,"rows_written":29950}
I20240503 08:12:51.346704 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1051371 bytes of WAL
I20240503 08:12:51.347358 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:51.347429 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000018 (ops 10561-11167)
I20240503 08:12:51.348570 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:51.348743 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008807
I20240503 08:12:51.452986 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.112s user 0.023s sys 0.016s Metrics: {"bytes_written":844951,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1226,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4974,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":54,"peak_mem_usage":0,"rows_written":29700}
I20240503 08:12:51.453310 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1058959 bytes of WAL
I20240503 08:12:51.453706 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:51.453789 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000019 (ops 10541-11123)
I20240503 08:12:51.454339 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:51.454718 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 292109 bytes on disk
I20240503 08:12:51.455164 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:51.455736 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008791
I20240503 08:12:51.905607 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.912s user 0.541s sys 0.048s Metrics: {"bytes_written":15795475,"cfile_cache_hit":13,"cfile_cache_hit_bytes":908124,"cfile_cache_miss":691,"cfile_cache_miss_bytes":40849696,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":771,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":5311,"drs_written":1,"lbm_read_time_us":25234,"lbm_reads_lt_1ms":711,"lbm_write_time_us":48203,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":687,"num_input_rowsets":2,"peak_mem_usage":9145278,"rows_written":568600,"spinlock_wait_cycles":6539392,"thread_start_us":297,"threads_started":6}
I20240503 08:12:51.905992 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:51.974463 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.068s user 0.042s sys 0.004s Metrics: {"bytes_written":1023206,"cfile_init":1,"dirs.queue_time_us":1125,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1144,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3255,"lbm_writes_lt_1ms":62,"peak_mem_usage":0,"rows_written":36000}
I20240503 08:12:51.974798 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1078493 bytes of WAL
I20240503 08:12:51.974925 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:51.974988 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000019 (ops 11319-11985)
I20240503 08:12:51.976538 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:51.976711 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6840189 bytes on disk
I20240503 08:12:51.977144 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20240503 08:12:51.977321 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008589
I20240503 08:12:52.490340 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 1.141s user 0.539s sys 0.062s Metrics: {"bytes_written":16141485,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1403607,"cfile_cache_miss":702,"cfile_cache_miss_bytes":41268209,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":18335,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":8081,"drs_written":1,"lbm_read_time_us":33961,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":722,"lbm_write_time_us":63220,"lbm_writes_1-10_ms":11,"lbm_writes_lt_1ms":694,"num_input_rowsets":2,"peak_mem_usage":9339278,"rows_written":581100,"spinlock_wait_cycles":93824,"thread_start_us":313,"threads_started":6}
I20240503 08:12:52.490696 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 6627190 bytes on disk
I20240503 08:12:52.491310 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:52.491492 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=4.913840
I20240503 08:12:52.565585 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.074s user 0.046s sys 0.007s Metrics: {"bytes_written":1342153,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":976,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2902,"lbm_writes_lt_1ms":75,"peak_mem_usage":0,"rows_written":47500}
I20240503 08:12:52.565801 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 2103614 bytes of WAL
I20240503 08:12:52.565913 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 2 log segments from log reader
I20240503 08:12:52.565955 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000019 (ops 11168-11779)
I20240503 08:12:52.566195 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000020 (ops 11780-12396)
I20240503 08:12:52.569058 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:52.569217 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008351
I20240503 08:12:52.705296 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 1.249s user 0.563s sys 0.040s Metrics: {"bytes_written":16181779,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1559888,"cfile_cache_miss":701,"cfile_cache_miss_bytes":41224772,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":9906,"drs_written":1,"lbm_read_time_us":28631,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":717,"lbm_write_time_us":60904,"lbm_writes_1-10_ms":9,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":697,"num_input_rowsets":2,"peak_mem_usage":9363903,"rows_written":582600,"spinlock_wait_cycles":9666816,"thread_start_us":358,"threads_started":6}
I20240503 08:12:52.705660 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=4.913840
I20240503 08:12:52.776012 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.070s user 0.047s sys 0.008s Metrics: {"bytes_written":1435628,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":897,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2927,"lbm_writes_lt_1ms":78,"peak_mem_usage":0,"rows_written":50800,"spinlock_wait_cycles":167680}
I20240503 08:12:52.776398 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008301
I20240503 08:12:52.833349 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.856s user 0.586s sys 0.041s Metrics: {"bytes_written":16790172,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1261503,"cfile_cache_miss":729,"cfile_cache_miss_bytes":43129915,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":14691,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1003,"drs_written":1,"lbm_read_time_us":23264,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":748,"lbm_write_time_us":46408,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":730,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":9715074,"rows_written":604600,"spinlock_wait_cycles":7213696,"thread_start_us":3090,"threads_started":6}
I20240503 08:12:52.833817 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=2.929465
I20240503 08:12:52.883141 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.048s user 0.036s sys 0.004s Metrics: {"bytes_written":895256,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":783,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2596,"lbm_writes_lt_1ms":57,"peak_mem_usage":0,"rows_written":31550}
I20240503 08:12:52.883409 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1049084 bytes of WAL
I20240503 08:12:52.883531 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:52.883585 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000020 (ops 11986-12613)
I20240503 08:12:52.883996 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:52.884130 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008234
I20240503 08:12:53.333151 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.764s user 0.586s sys 0.052s Metrics: {"bytes_written":17450628,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1448808,"cfile_cache_miss":756,"cfile_cache_miss_bytes":44694412,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":8179,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":2330,"drs_written":1,"lbm_read_time_us":23101,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":779,"lbm_write_time_us":41968,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":758,"num_input_rowsets":2,"peak_mem_usage":10093531,"rows_written":628600,"spinlock_wait_cycles":1179392,"thread_start_us":306,"threads_started":6}
I20240503 08:12:53.333588 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:53.379910 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.046s user 0.024s sys 0.004s Metrics: {"bytes_written":533002,"cfile_init":1,"dirs.queue_time_us":19,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3934,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":18750}
I20240503 08:12:53.380848 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7357379 bytes on disk
I20240503 08:12:53.384375 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20240503 08:12:53.384838 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008155
I20240503 08:12:53.646414 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.870s user 0.597s sys 0.052s Metrics: {"bytes_written":17585180,"cfile_cache_hit":13,"cfile_cache_hit_bytes":988175,"cfile_cache_miss":763,"cfile_cache_miss_bytes":45509187,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":10655,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":22662,"lbm_reads_lt_1ms":791,"lbm_write_time_us":44303,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":765,"num_input_rowsets":2,"peak_mem_usage":10168223,"rows_written":633400,"spinlock_wait_cycles":1235200,"thread_start_us":308,"threads_started":6}
I20240503 08:12:53.646826 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 2136043 bytes of WAL
I20240503 08:12:53.647009 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 2 log segments from log reader
I20240503 08:12:53.647116 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000020 (ops 11124-11707)
I20240503 08:12:53.647204 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000021 (ops 11708-12299)
I20240503 08:12:53.647960 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:53.648145 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7233632 bytes on disk
I20240503 08:12:53.648355 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:53.648525 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:53.699966 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.816s user 0.612s sys 0.062s Metrics: {"bytes_written":17663696,"cfile_cache_hit":13,"cfile_cache_hit_bytes":935470,"cfile_cache_miss":771,"cfile_cache_miss_bytes":45788300,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":10651,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1058,"drs_written":1,"lbm_read_time_us":27498,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":798,"lbm_write_time_us":43270,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":770,"num_input_rowsets":2,"peak_mem_usage":10232175,"rows_written":636150,"spinlock_wait_cycles":497024,"thread_start_us":345,"threads_started":6}
I20240503 08:12:53.700335 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:53.718025 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.069s user 0.023s sys 0.007s Metrics: {"bytes_written":564475,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1931,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":19800,"spinlock_wait_cycles":154880}
I20240503 08:12:53.718276 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1056915 bytes of WAL
I20240503 08:12:53.718420 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:53.718469 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000022 (ops 12300-12881)
I20240503 08:12:53.718956 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:53.719097 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 191830 bytes on disk
I20240503 08:12:53.719924 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:12:53.720084 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008092
I20240503 08:12:53.741899 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.040s user 0.023s sys 0.003s Metrics: {"bytes_written":504537,"cfile_init":1,"dirs.queue_time_us":19,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2035,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17650}
I20240503 08:12:53.742264 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.008084
I20240503 08:12:54.169935 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.785s user 0.612s sys 0.046s Metrics: {"bytes_written":17968883,"cfile_cache_hit":11,"cfile_cache_hit_bytes":588612,"cfile_cache_miss":787,"cfile_cache_miss_bytes":46969342,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1050,"drs_written":1,"lbm_read_time_us":27217,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":806,"lbm_write_time_us":39631,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":782,"num_input_rowsets":2,"peak_mem_usage":10426975,"rows_written":647350,"spinlock_wait_cycles":7424,"thread_start_us":286,"threads_started":6}
I20240503 08:12:54.170310 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:54.189733 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.019s user 0.018s sys 0.000s Metrics: {"bytes_written":412136,"cfile_init":1,"dirs.queue_time_us":36,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14300}
I20240503 08:12:54.190008 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1054119 bytes of WAL
I20240503 08:12:54.190165 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:54.190241 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000021 (ops 12397-12988)
I20240503 08:12:54.190694 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:54.190830 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007903
I20240503 08:12:54.269343 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.527s user 0.447s sys 0.055s Metrics: {"bytes_written":18145219,"cfile_cache_hit":11,"cfile_cache_hit_bytes":585519,"cfile_cache_miss":795,"cfile_cache_miss_bytes":47457373,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1106,"drs_written":1,"lbm_read_time_us":19737,"lbm_reads_lt_1ms":823,"lbm_write_time_us":31345,"lbm_writes_lt_1ms":791,"num_input_rowsets":2,"peak_mem_usage":10532881,"rows_written":653800,"spinlock_wait_cycles":1664,"thread_start_us":210,"threads_started":5}
I20240503 08:12:54.269697 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7471501 bytes on disk
I20240503 08:12:54.269901 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:54.447358 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.727s user 0.572s sys 0.072s Metrics: {"bytes_written":18126862,"cfile_cache_hit":11,"cfile_cache_hit_bytes":591619,"cfile_cache_miss":793,"cfile_cache_miss_bytes":47401621,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":300,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":944,"drs_written":1,"lbm_read_time_us":23888,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":810,"lbm_write_time_us":39382,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":789,"num_input_rowsets":2,"peak_mem_usage":10520981,"rows_written":653200,"spinlock_wait_cycles":768,"thread_start_us":348,"threads_started":6}
I20240503 08:12:54.451124 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7464286 bytes on disk
I20240503 08:12:54.451603 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:54.548033 28083 raft_consensus.cc:979] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: : Instructing follower 9bd13ada0adf49b994fa8c4a425744f1 to start an election
I20240503 08:12:54.548133 28083 raft_consensus.cc:1067] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 LEADER]: Signalling peer 9bd13ada0adf49b994fa8c4a425744f1 to start an election
I20240503 08:12:54.548352 27500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0"
dest_uuid: "9bd13ada0adf49b994fa8c4a425744f1"
from {username='slave'} at 127.0.0.1:56626
I20240503 08:12:54.548451 27500 raft_consensus.cc:482] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 08:12:54.548483 27500 raft_consensus.cc:3037] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:54.549182 27500 raft_consensus.cc:504] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:54.549355 27500 leader_election.cc:288] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [CANDIDATE]: Term 2 election: Requested vote from peers f83e30770df64414b2a76bb20ff27be6 (127.16.212.194:42371), 810dfe154e0d4edfa5625eea399613c4 (127.16.212.195:37211)
I20240503 08:12:54.552744 27651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "9bd13ada0adf49b994fa8c4a425744f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 13234 } } ignore_live_leader: true dest_uuid: "810dfe154e0d4edfa5625eea399613c4"
I20240503 08:12:54.552762 27576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c30cc6bf11a44b987f50663b2f102c0" candidate_uuid: "9bd13ada0adf49b994fa8c4a425744f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 13234 } } ignore_live_leader: true dest_uuid: "f83e30770df64414b2a76bb20ff27be6"
I20240503 08:12:54.552851 27651 raft_consensus.cc:3037] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:54.552871 27576 raft_consensus.cc:3032] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 08:12:54.552907 27576 raft_consensus.cc:729] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f83e30770df64414b2a76bb20ff27be6, State: Running, Role: LEADER
I20240503 08:12:54.552959 27576 consensus_queue.cc:260] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13234, Committed index: 13234, Last appended: 1.13234, Last appended by leader: 13234, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:54.553036 27576 raft_consensus.cc:3037] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 08:12:54.553740 27576 raft_consensus.cc:2454] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd13ada0adf49b994fa8c4a425744f1 in term 2.
I20240503 08:12:54.553741 27651 raft_consensus.cc:2454] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bd13ada0adf49b994fa8c4a425744f1 in term 2.
I20240503 08:12:54.553957 27463 leader_election.cc:302] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bd13ada0adf49b994fa8c4a425744f1, f83e30770df64414b2a76bb20ff27be6; no voters:
I20240503 08:12:54.554150 28135 raft_consensus.cc:2781] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 08:12:54.554266 28135 raft_consensus.cc:686] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 2 LEADER]: Becoming Leader. State: Replica: 9bd13ada0adf49b994fa8c4a425744f1, State: Running, Role: LEADER
I20240503 08:12:54.554327 28135 consensus_queue.cc:237] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13234, Committed index: 13234, Last appended: 1.13234, Last appended by leader: 13234, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } }
I20240503 08:12:54.554800 27396 catalog_manager.cc:5484] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 reported cstate change: term changed from 1 to 2, leader changed from f83e30770df64414b2a76bb20ff27be6 (127.16.212.194) to 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193). New cstate: current_term: 2 leader_uuid: "9bd13ada0adf49b994fa8c4a425744f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } health_report { overall_health: UNKNOWN } } }
I20240503 08:12:54.561125 27651 raft_consensus.cc:1261] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 2 FOLLOWER]: Refusing update from remote peer 9bd13ada0adf49b994fa8c4a425744f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 13234. Preceding OpId from leader: term: 2 index: 13236. (index mismatch)
I20240503 08:12:54.561125 27576 raft_consensus.cc:1261] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 2 FOLLOWER]: Refusing update from remote peer 9bd13ada0adf49b994fa8c4a425744f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 13234. Preceding OpId from leader: term: 2 index: 13236. (index mismatch)
I20240503 08:12:54.561383 28135 consensus_queue.cc:1035] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13235, Last known committed idx: 13234, Time since last communication: 0.000s
I20240503 08:12:54.561641 28135 consensus_queue.cc:1035] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13235, Last known committed idx: 13234, Time since last communication: 0.000s
I20240503 08:12:54.589048 17235 test_util.cc:268] Using random seed: 1231334003
I20240503 08:12:54.595124 17235 test_workload.cc:401] TestWorkload: Skipping table creation because table test-workload already exists
I20240503 08:12:54.644971 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:54.665871 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:54.673620 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.028s user 0.015s sys 0.000s Metrics: {"bytes_written":281601,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":931,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":703,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:54.673939 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007818
I20240503 08:12:54.739450 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.549s user 0.504s sys 0.043s Metrics: {"bytes_written":18361666,"cfile_cache_hit":10,"cfile_cache_hit_bytes":532476,"cfile_cache_miss":804,"cfile_cache_miss_bytes":48091798,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":943,"drs_written":1,"lbm_read_time_us":19250,"lbm_reads_lt_1ms":832,"lbm_write_time_us":27329,"lbm_writes_lt_1ms":801,"num_input_rowsets":2,"peak_mem_usage":10663795,"rows_written":661650,"spinlock_wait_cycles":10368}
I20240503 08:12:54.739784 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7561798 bytes on disk
I20240503 08:12:54.740001 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:54.751040 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.085s user 0.016s sys 0.000s Metrics: {"bytes_written":279130,"cfile_init":1,"dirs.queue_time_us":8304,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1895,"drs_written":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1440,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700,"spinlock_wait_cycles":151424}
I20240503 08:12:54.752547 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1053366 bytes of WAL
I20240503 08:12:54.752692 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:54.752771 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000021 (ops 12614-13219)
I20240503 08:12:54.753309 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:54.754519 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007812
I20240503 08:12:54.871866 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.000000
I20240503 08:12:54.897166 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":280370,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":945,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":625,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:12:54.897655 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007826
I20240503 08:12:55.392817 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.719s user 0.604s sys 0.052s Metrics: {"bytes_written":18400348,"cfile_cache_hit":13,"cfile_cache_hit_bytes":795426,"cfile_cache_miss":803,"cfile_cache_miss_bytes":47933788,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":13653,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":825,"drs_written":1,"lbm_read_time_us":26863,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":824,"lbm_write_time_us":33241,"lbm_writes_lt_1ms":803,"num_input_rowsets":2,"peak_mem_usage":10690231,"rows_written":663000,"spinlock_wait_cycles":238720,"thread_start_us":275,"threads_started":6}
I20240503 08:12:55.393108 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7578660 bytes on disk
I20240503 08:12:55.393312 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:55.393460 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:55.466985 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.073s user 0.026s sys 0.004s Metrics: {"bytes_written":620817,"cfile_init":1,"dirs.queue_time_us":996,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1019,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5784,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":21800,"spinlock_wait_cycles":153856}
I20240503 08:12:55.467320 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1074860 bytes of WAL
I20240503 08:12:55.467442 27543 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:55.467530 27543 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000023 (ops 12882-13520)
I20240503 08:12:55.468083 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:55.468247 27603 maintenance_manager.cc:419] P f83e30770df64414b2a76bb20ff27be6: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007767
I20240503 08:12:55.561507 27386 maintenance_manager.cc:419] P e4539b3eedcc4ea4aa3566df9db5aa81: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.000728
I20240503 08:12:55.570338 27385 maintenance_manager.cc:643] P e4539b3eedcc4ea4aa3566df9db5aa81: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.000s sys 0.001s Metrics: {"bytes_written":430,"delete_count":0,"lbm_write_time_us":42,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":61,"threads_started":1,"update_count":1,"wal-append.queue_time_us":7429}
I20240503 08:12:55.570513 27386 maintenance_manager.cc:419] P e4539b3eedcc4ea4aa3566df9db5aa81: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.196886
I20240503 08:12:55.573015 27385 maintenance_manager.cc:643] P e4539b3eedcc4ea4aa3566df9db5aa81: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.002s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":2100,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":17,"lbm_write_time_us":94,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":139,"threads_started":2,"update_count":1}
I20240503 08:12:55.589888 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.835s user 0.631s sys 0.039s Metrics: {"bytes_written":18416373,"cfile_cache_hit":13,"cfile_cache_hit_bytes":792732,"cfile_cache_miss":803,"cfile_cache_miss_bytes":47973196,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":11301,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":787,"drs_written":1,"lbm_read_time_us":29696,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":825,"lbm_write_time_us":54103,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":802,"num_input_rowsets":2,"peak_mem_usage":10698456,"rows_written":663500,"spinlock_wait_cycles":7309568,"thread_start_us":384,"threads_started":6}
I20240503 08:12:55.590240 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7584146 bytes on disk
I20240503 08:12:55.592020 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:55.592471 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:55.628757 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.036s user 0.028s sys 0.004s Metrics: {"bytes_written":769784,"cfile_init":1,"dirs.queue_time_us":26,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1481,"lbm_writes_lt_1ms":51,"peak_mem_usage":0,"rows_written":27050}
I20240503 08:12:55.629025 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 265245 bytes on disk
I20240503 08:12:55.629253 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:12:55.629439 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007707
I20240503 08:12:55.782505 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.885s user 0.607s sys 0.060s Metrics: {"bytes_written":18629638,"cfile_cache_hit":14,"cfile_cache_hit_bytes":949764,"cfile_cache_miss":814,"cfile_cache_miss_bytes":48398406,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":325,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":14655,"drs_written":1,"lbm_read_time_us":30996,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":836,"lbm_write_time_us":41306,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":812,"num_input_rowsets":2,"peak_mem_usage":10826772,"rows_written":671400,"spinlock_wait_cycles":108672,"thread_start_us":1110,"threads_started":6}
I20240503 08:12:55.783743 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7675716 bytes on disk
I20240503 08:12:55.783931 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:55.784097 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.937278
I20240503 08:12:55.836478 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.052s user 0.025s sys 0.004s Metrics: {"bytes_written":667921,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":7361,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4244,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":46,"peak_mem_usage":0,"rows_written":23450}
I20240503 08:12:55.836791 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1051284 bytes of WAL
I20240503 08:12:55.836963 27467 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:55.837085 27467 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000022 (ops 12989-13600)
I20240503 08:12:55.837906 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:55.838191 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007622
I20240503 08:12:55.857163 17235 tablet_server.cc:178] TabletServer@127.16.212.194:0 shutting down...
I20240503 08:12:56.241816 27543 maintenance_manager.cc:643] P f83e30770df64414b2a76bb20ff27be6: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.773s user 0.602s sys 0.086s Metrics: {"bytes_written":18999541,"cfile_cache_hit":11,"cfile_cache_hit_bytes":597699,"cfile_cache_miss":833,"cfile_cache_miss_bytes":49714949,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":250,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":820,"drs_written":1,"lbm_read_time_us":21161,"lbm_reads_lt_1ms":857,"lbm_write_time_us":45059,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":826,"num_input_rowsets":2,"peak_mem_usage":11032444,"rows_written":684800,"spinlock_wait_cycles":6144,"thread_start_us":317,"threads_started":6}
I20240503 08:12:56.242313 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:56.242481 17235 tablet_replica.cc:330] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6: stopping tablet replica
I20240503 08:12:56.242525 17235 raft_consensus.cc:2229] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:56.242586 17235 raft_consensus.cc:2258] T 1c30cc6bf11a44b987f50663b2f102c0 P f83e30770df64414b2a76bb20ff27be6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 08:12:56.252318 27463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.212.194:42371: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 08:12:56.252709 27463 consensus_peers.cc:485] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 -> Peer f83e30770df64414b2a76bb20ff27be6 (127.16.212.194:42371): Couldn't send request to peer f83e30770df64414b2a76bb20ff27be6. Status: Network error: Client connection negotiation failed: client connection to 127.16.212.194:42371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:56.258260 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.420s user 0.379s sys 0.033s Metrics: {"bytes_written":19276901,"cfile_cache_hit":10,"cfile_cache_hit_bytes":532868,"cfile_cache_miss":844,"cfile_cache_miss_bytes":50516576,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":967,"drs_written":1,"lbm_read_time_us":18181,"lbm_reads_lt_1ms":868,"lbm_write_time_us":23366,"lbm_writes_lt_1ms":841,"num_input_rowsets":2,"peak_mem_usage":11194095,"rows_written":694850,"spinlock_wait_cycles":384}
I20240503 08:12:56.258947 17235 tablet_server.cc:194] TabletServer@127.16.212.194:0 shutdown complete.
I20240503 08:12:56.259981 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7949038 bytes on disk
I20240503 08:12:56.260186 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:56.295504 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:56.298429 28246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:56.299976 17235 server_base.cc:1034] running on GCE node
W20240503 08:12:56.300477 28245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:56.301353 28248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:56.301576 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:56.301623 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:56.301648 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723976301649 us; error 0 us; skew 500 ppm
I20240503 08:12:56.302702 17235 webserver.cc:460] Webserver started at http://127.16.212.196:38175/ using document root <none> and password file <none>
I20240503 08:12:56.302801 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:56.302865 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:56.302917 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:56.303829 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-3-root/instance:
uuid: "0a4aa784729d4fa49e572e0af6df8b05"
format_stamp: "Formatted at 2024-05-03 08:12:56 on dist-test-slave-zl2t"
I20240503 08:12:56.305141 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 08:12:56.306396 28253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.306711 17235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:56.306797 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-3-root
uuid: "0a4aa784729d4fa49e572e0af6df8b05"
format_stamp: "Formatted at 2024-05-03 08:12:56 on dist-test-slave-zl2t"
I20240503 08:12:56.306915 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:56.312201 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.683s user 0.613s sys 0.068s Metrics: {"bytes_written":19157761,"cfile_cache_hit":10,"cfile_cache_hit_bytes":532822,"cfile_cache_miss":838,"cfile_cache_miss_bytes":50199998,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":292,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1066,"drs_written":1,"lbm_read_time_us":20533,"lbm_reads_lt_1ms":858,"lbm_write_time_us":32100,"lbm_writes_lt_1ms":835,"num_input_rowsets":2,"peak_mem_usage":11120620,"rows_written":690550,"spinlock_wait_cycles":896,"thread_start_us":340,"threads_started":6}
I20240503 08:12:56.312404 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7898702 bytes on disk
I20240503 08:12:56.312577 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:56.330034 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:56.330274 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:56.330451 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:56.330716 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:56.330752 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.330786 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:56.330832 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:56.333634 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.196:35323
I20240503 08:12:56.334367 28321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.196:35323 every 8 connection(s)
I20240503 08:12:56.342231 28322 heartbeater.cc:347] Connected to a master server at 127.16.212.254:40439
I20240503 08:12:56.342317 28322 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:56.342425 28322 heartbeater.cc:511] Master 127.16.212.254:40439 requested a full tablet report, sending...
I20240503 08:12:56.342998 27395 ts_manager.cc:194] Registered new tserver with Master: 0a4aa784729d4fa49e572e0af6df8b05 (127.16.212.196:35323)
I20240503 08:12:56.343394 27395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40922
I20240503 08:12:56.629082 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=0.972434
I20240503 08:12:56.635572 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"bytes_written":162194,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":979,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":382,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":5600}
I20240503 08:12:56.635777 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling LogGCOp(1c30cc6bf11a44b987f50663b2f102c0): free 1048056 bytes of WAL
I20240503 08:12:56.635905 27618 log_reader.cc:386] T 1c30cc6bf11a44b987f50663b2f102c0: removed 1 log segments from log reader
I20240503 08:12:56.635938 27618 log.cc:1076] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/1c30cc6bf11a44b987f50663b2f102c0/wal-000000022 (ops 13220-13855)
I20240503 08:12:56.636301 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: LogGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:56.636411 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007450
I20240503 08:12:56.841699 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=0.242942
I20240503 08:12:56.845055 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: FlushMRSOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":38307,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":945,"drs_written":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":268,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":1300}
I20240503 08:12:56.845366 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 11308 bytes on disk
I20240503 08:12:56.845615 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:56.845813 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0): perf score=1.007228
I20240503 08:12:57.030934 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.394s user 0.358s sys 0.036s Metrics: {"bytes_written":19311254,"cfile_cache_hit":10,"cfile_cache_hit_bytes":374462,"cfile_cache_miss":848,"cfile_cache_miss_bytes":50800608,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":2153,"drs_written":1,"lbm_read_time_us":21984,"lbm_reads_lt_1ms":872,"lbm_write_time_us":23478,"lbm_writes_lt_1ms":842,"num_input_rowsets":2,"peak_mem_usage":11233086,"rows_written":696150,"thread_start_us":205,"threads_started":5}
I20240503 08:12:57.031265 27678 maintenance_manager.cc:419] P 810dfe154e0d4edfa5625eea399613c4: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7964262 bytes on disk
I20240503 08:12:57.031417 27618 maintenance_manager.cc:643] P 810dfe154e0d4edfa5625eea399613c4: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:57.192719 28327 consensus_queue.cc:579] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [LEADER]: Leader has been unable to successfully communicate with peer f83e30770df64414b2a76bb20ff27be6 for more than 1 seconds (1.336s)
I20240503 08:12:57.193835 27500 consensus_queue.cc:237] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13955, Committed index: 13955, Last appended: 2.13955, Last appended by leader: 13234, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13956 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } } peers { permanent_uuid: "0a4aa784729d4fa49e572e0af6df8b05" member_type: NON_VOTER last_known_addr { host: "127.16.212.196" port: 35323 } attrs { promote: true } }
I20240503 08:12:57.194706 27651 raft_consensus.cc:1261] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 2 FOLLOWER]: Refusing update from remote peer 9bd13ada0adf49b994fa8c4a425744f1: Log matching property violated. Preceding OpId in replica: term: 2 index: 13955. Preceding OpId from leader: term: 2 index: 13956. (index mismatch)
W20240503 08:12:57.194769 27463 consensus_peers.cc:485] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 -> Peer f83e30770df64414b2a76bb20ff27be6 (127.16.212.194:42371): Couldn't send request to peer f83e30770df64414b2a76bb20ff27be6. Status: Network error: Client connection negotiation failed: client connection to 127.16.212.194:42371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:57.195235 28333 consensus_queue.cc:1035] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13956, Last known committed idx: 13955, Time since last communication: 0.000s
I20240503 08:12:57.195850 28336 raft_consensus.cc:2932] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 2 LEADER]: Committing config change with OpId 2.13956: config changed from index -1 to 13956, NON_VOTER 0a4aa784729d4fa49e572e0af6df8b05 (127.16.212.196) added. New config: { opid_index: 13956 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } } peers { permanent_uuid: "0a4aa784729d4fa49e572e0af6df8b05" member_type: NON_VOTER last_known_addr { host: "127.16.212.196" port: 35323 } attrs { promote: true } } }
I20240503 08:12:57.195938 27651 raft_consensus.cc:2932] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 2 FOLLOWER]: Committing config change with OpId 2.13956: config changed from index -1 to 13956, NON_VOTER 0a4aa784729d4fa49e572e0af6df8b05 (127.16.212.196) added. New config: { opid_index: 13956 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } } peers { permanent_uuid: "0a4aa784729d4fa49e572e0af6df8b05" member_type: NON_VOTER last_known_addr { host: "127.16.212.196" port: 35323 } attrs { promote: true } } }
I20240503 08:12:57.197093 27382 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1c30cc6bf11a44b987f50663b2f102c0 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 08:12:57.197436 27393 catalog_manager.cc:5484] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 reported cstate change: config changed from index -1 to 13956, NON_VOTER 0a4aa784729d4fa49e572e0af6df8b05 (127.16.212.196) added. New cstate: current_term: 2 leader_uuid: "9bd13ada0adf49b994fa8c4a425744f1" committed_config { opid_index: 13956 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a4aa784729d4fa49e572e0af6df8b05" member_type: NON_VOTER last_known_addr { host: "127.16.212.196" port: 35323 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 08:12:57.199030 27465 consensus_peers.cc:485] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 -> Peer 0a4aa784729d4fa49e572e0af6df8b05 (127.16.212.196:35323): Couldn't send request to peer 0a4aa784729d4fa49e572e0af6df8b05. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c30cc6bf11a44b987f50663b2f102c0. This is attempt 1: this message will repeat every 5th retry.
I20240503 08:12:57.232867 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: CompactRowSetsOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.387s user 0.327s sys 0.060s Metrics: {"bytes_written":19311254,"cfile_cache_hit":10,"cfile_cache_hit_bytes":294494,"cfile_cache_miss":850,"cfile_cache_miss_bytes":50893612,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":568,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1836,"drs_written":1,"lbm_read_time_us":18399,"lbm_reads_lt_1ms":870,"lbm_write_time_us":23426,"lbm_writes_lt_1ms":842,"mutex_wait_us":2,"num_input_rowsets":2,"peak_mem_usage":11238381,"rows_written":696150,"spinlock_wait_cycles":1408,"thread_start_us":408,"threads_started":5}
I20240503 08:12:57.233250 27527 maintenance_manager.cc:419] P 9bd13ada0adf49b994fa8c4a425744f1: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7964262 bytes on disk
I20240503 08:12:57.233464 27467 maintenance_manager.cc:643] P 9bd13ada0adf49b994fa8c4a425744f1: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:57.344262 28322 heartbeater.cc:503] Master 127.16.212.254:40439 was elected leader, sending a full tablet report...
I20240503 08:12:57.648862 28349 ts_tablet_manager.cc:902] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: Initiating tablet copy from peer 9bd13ada0adf49b994fa8c4a425744f1 (127.16.212.193:40597)
I20240503 08:12:57.649202 28349 tablet_copy_client.cc:308] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: tablet copy: Beginning tablet copy session from remote peer at address 127.16.212.193:40597
I20240503 08:12:57.651887 27510 tablet_copy_service.cc:140] P 9bd13ada0adf49b994fa8c4a425744f1: Received BeginTabletCopySession request for tablet 1c30cc6bf11a44b987f50663b2f102c0 from peer 0a4aa784729d4fa49e572e0af6df8b05 ({username='slave'} at 127.0.0.1:57794)
I20240503 08:12:57.651947 27510 tablet_copy_service.cc:161] P 9bd13ada0adf49b994fa8c4a425744f1: Beginning new tablet copy session on tablet 1c30cc6bf11a44b987f50663b2f102c0 from peer 0a4aa784729d4fa49e572e0af6df8b05 at {username='slave'} at 127.0.0.1:57794: session id = 0a4aa784729d4fa49e572e0af6df8b05-1c30cc6bf11a44b987f50663b2f102c0
I20240503 08:12:57.652536 27510 tablet_copy_source_session.cc:205] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 08:12:57.652962 28349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c30cc6bf11a44b987f50663b2f102c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:57.653981 28349 tablet_copy_client.cc:783] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: tablet copy: Starting download of 5 data blocks...
I20240503 08:12:57.705864 28349 tablet_copy_client.cc:647] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: tablet copy: Starting download of 1 WAL segments...
I20240503 08:12:57.707741 28349 tablet_copy_client.cc:523] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 08:12:57.708554 28349 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: Bootstrap starting.
I20240503 08:12:57.735571 28349 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: Bootstrap replayed 1/1 log segments. Stats: ops{read=356 overwritten=0 applied=356 ignored=355} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 08:12:57.735812 28349 tablet_bootstrap.cc:492] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: Bootstrap complete.
I20240503 08:12:57.736011 28349 ts_tablet_manager.cc:1367] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.005s
I20240503 08:12:57.736150 28349 raft_consensus.cc:348] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13956 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } } peers { permanent_uuid: "0a4aa784729d4fa49e572e0af6df8b05" member_type: NON_VOTER last_known_addr { host: "127.16.212.196" port: 35323 } attrs { promote: true } }
I20240503 08:12:57.736222 28349 raft_consensus.cc:729] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0a4aa784729d4fa49e572e0af6df8b05, State: Initialized, Role: LEARNER
I20240503 08:12:57.736245 28349 consensus_queue.cc:260] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13956, Last appended: 2.13956, Last appended by leader: 13956, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13956 OBSOLETE_local: false peers { permanent_uuid: "9bd13ada0adf49b994fa8c4a425744f1" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 40597 } } peers { permanent_uuid: "f83e30770df64414b2a76bb20ff27be6" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 42371 } } peers { permanent_uuid: "810dfe154e0d4edfa5625eea399613c4" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 37211 } } peers { permanent_uuid: "0a4aa784729d4fa49e572e0af6df8b05" member_type: NON_VOTER last_known_addr { host: "127.16.212.196" port: 35323 } attrs { promote: true } }
I20240503 08:12:57.736385 28349 ts_tablet_manager.cc:1398] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20240503 08:12:57.736672 27510 tablet_copy_service.cc:342] P 9bd13ada0adf49b994fa8c4a425744f1: Request end of tablet copy session 0a4aa784729d4fa49e572e0af6df8b05-1c30cc6bf11a44b987f50663b2f102c0 received from {username='slave'} at 127.0.0.1:57794
I20240503 08:12:57.736730 27510 tablet_copy_service.cc:433] P 9bd13ada0adf49b994fa8c4a425744f1: ending tablet copy session 0a4aa784729d4fa49e572e0af6df8b05-1c30cc6bf11a44b987f50663b2f102c0 on tablet 1c30cc6bf11a44b987f50663b2f102c0 with peer 0a4aa784729d4fa49e572e0af6df8b05
I20240503 08:12:57.745059 28323 maintenance_manager.cc:419] P 0a4aa784729d4fa49e572e0af6df8b05: Scheduling UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0): 7964262 bytes on disk
I20240503 08:12:57.745360 28263 maintenance_manager.cc:643] P 0a4aa784729d4fa49e572e0af6df8b05: UndoDeltaBlockGCOp(1c30cc6bf11a44b987f50663b2f102c0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:58.129874 28296 raft_consensus.cc:1203] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13955->[2.13956-2.13956] Dedup: 2.13956->[]
I20240503 08:12:58.203517 27386 maintenance_manager.cc:419] P e4539b3eedcc4ea4aa3566df9db5aa81: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.000728
I20240503 08:12:58.204761 27385 maintenance_manager.cc:643] P e4539b3eedcc4ea4aa3566df9db5aa81: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":477,"delete_count":0,"lbm_write_time_us":53,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":53,"threads_started":1,"update_count":1,"wal-append.queue_time_us":87}
I20240503 08:12:58.204898 27386 maintenance_manager.cc:419] P e4539b3eedcc4ea4aa3566df9db5aa81: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.218407
I20240503 08:12:58.206197 27385 maintenance_manager.cc:643] P e4539b3eedcc4ea4aa3566df9db5aa81: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":2147,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":94,"lbm_read_time_us":42,"lbm_reads_lt_1ms":17,"lbm_write_time_us":59,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":77,"threads_started":2,"update_count":1}
I20240503 08:12:58.359112 17235 tablet_server.cc:178] TabletServer@127.16.212.193:0 shutting down...
I20240503 08:12:58.362409 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:58.362650 17235 tablet_replica.cc:330] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1: stopping tablet replica
I20240503 08:12:58.362730 17235 raft_consensus.cc:2229] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 2 LEADER]: Raft consensus shutting down.
I20240503 08:12:58.362838 17235 raft_consensus.cc:2258] T 1c30cc6bf11a44b987f50663b2f102c0 P 9bd13ada0adf49b994fa8c4a425744f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.374958 17235 tablet_server.cc:194] TabletServer@127.16.212.193:0 shutdown complete.
I20240503 08:12:58.376149 17235 tablet_server.cc:178] TabletServer@127.16.212.195:0 shutting down...
I20240503 08:12:58.379845 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:58.380046 17235 tablet_replica.cc:330] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4: stopping tablet replica
I20240503 08:12:58.380115 17235 raft_consensus.cc:2229] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 08:12:58.380216 17235 raft_consensus.cc:2258] T 1c30cc6bf11a44b987f50663b2f102c0 P 810dfe154e0d4edfa5625eea399613c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.391996 17235 tablet_server.cc:194] TabletServer@127.16.212.195:0 shutdown complete.
I20240503 08:12:58.393066 17235 tablet_server.cc:178] TabletServer@127.16.212.196:0 shutting down...
I20240503 08:12:58.395696 17235 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 08:12:58.395838 17235 tablet_replica.cc:330] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05: stopping tablet replica
I20240503 08:12:58.395892 17235 raft_consensus.cc:2229] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05 [term 2 LEARNER]: Raft consensus shutting down.
I20240503 08:12:58.395943 17235 raft_consensus.cc:2258] T 1c30cc6bf11a44b987f50663b2f102c0 P 0a4aa784729d4fa49e572e0af6df8b05 [term 2 LEARNER]: Raft consensus is shut down!
I20240503 08:12:58.407595 17235 tablet_server.cc:194] TabletServer@127.16.212.196:0 shutdown complete.
I20240503 08:12:58.408808 17235 master.cc:537] Master@127.16.212.254:40439 shutting down...
I20240503 08:12:58.412736 17235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 1 LEADER]: Raft consensus shutting down.
I20240503 08:12:58.412824 17235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 08:12:58.412844 17235 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4539b3eedcc4ea4aa3566df9db5aa81: stopping tablet replica
I20240503 08:12:58.424715 17235 master.cc:552] Master@127.16.212.254:40439 shutdown complete.
[ OK ] ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps/0 (24833 ms)
[ RUN ] ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps/1
I20240503 08:12:58.455327 17235 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.16.212.254:39047
I20240503 08:12:58.455533 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:58.458293 28363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:58.459625 28362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:58.460685 28365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.461851 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:58.461978 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:58.462021 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:58.462041 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723978462042 us; error 0 us; skew 500 ppm
I20240503 08:12:58.468420 17235 webserver.cc:460] Webserver started at http://127.16.212.254:45873/ using document root <none> and password file <none>
I20240503 08:12:58.468518 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:58.468552 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:58.468595 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:58.468899 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/master-0-root/instance:
uuid: "a325bde86de4459292246df81b755452"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.469837 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:58.470382 28370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.470561 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.470638 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/master-0-root
uuid: "a325bde86de4459292246df81b755452"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.470687 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:58.476689 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:58.476861 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:58.479363 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.254:39047
I20240503 08:12:58.481268 28422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.254:39047 every 8 connection(s)
I20240503 08:12:58.482312 28423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.485994 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452: Bootstrap starting.
I20240503 08:12:58.486338 28423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.486903 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452: No bootstrap required, opened a new log
I20240503 08:12:58.487016 28423 raft_consensus.cc:348] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a325bde86de4459292246df81b755452" member_type: VOTER }
I20240503 08:12:58.487082 28423 raft_consensus.cc:374] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.487095 28423 raft_consensus.cc:729] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a325bde86de4459292246df81b755452, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.487149 28423 consensus_queue.cc:260] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a325bde86de4459292246df81b755452" member_type: VOTER }
I20240503 08:12:58.487185 28423 raft_consensus.cc:388] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 08:12:58.487202 28423 raft_consensus.cc:482] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 08:12:58.487227 28423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.487761 28423 raft_consensus.cc:504] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a325bde86de4459292246df81b755452" member_type: VOTER }
I20240503 08:12:58.487818 28423 leader_election.cc:302] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a325bde86de4459292246df81b755452; no voters:
I20240503 08:12:58.487912 28423 leader_election.cc:288] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 08:12:58.488006 28426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.488052 28423 sys_catalog.cc:564] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 08:12:58.488148 28426 raft_consensus.cc:686] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [term 1 LEADER]: Becoming Leader. State: Replica: a325bde86de4459292246df81b755452, State: Running, Role: LEADER
I20240503 08:12:58.488205 28426 consensus_queue.cc:237] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a325bde86de4459292246df81b755452" member_type: VOTER }
I20240503 08:12:58.488369 28426 sys_catalog.cc:455] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a325bde86de4459292246df81b755452. Latest consensus state: current_term: 1 leader_uuid: "a325bde86de4459292246df81b755452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a325bde86de4459292246df81b755452" member_type: VOTER } }
I20240503 08:12:58.488396 28428 sys_catalog.cc:455] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a325bde86de4459292246df81b755452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a325bde86de4459292246df81b755452" member_type: VOTER } }
I20240503 08:12:58.488512 28426 sys_catalog.cc:458] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:58.488607 28428 sys_catalog.cc:458] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452 [sys.catalog]: This master's current role is: LEADER
I20240503 08:12:58.488797 28436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 08:12:58.488922 28436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 08:12:58.489400 17235 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 08:12:58.489442 28436 catalog_manager.cc:1347] Generated new cluster ID: 2bd60e4d2bbd4c4fbd14bfaa09b819e6
I20240503 08:12:58.489485 28436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 08:12:58.505841 28436 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 08:12:58.506311 28436 catalog_manager.cc:1504] Loading token signing keys...
I20240503 08:12:58.517542 28436 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a325bde86de4459292246df81b755452: Generated new TSK 0
I20240503 08:12:58.517644 28436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 08:12:58.521342 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:58.523881 28445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:58.525048 28444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 08:12:58.526077 28447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.527154 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:58.527251 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:58.527289 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:58.527319 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723978527320 us; error 0 us; skew 500 ppm
I20240503 08:12:58.528049 17235 webserver.cc:460] Webserver started at http://127.16.212.193:37591/ using document root <none> and password file <none>
I20240503 08:12:58.528138 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:58.528189 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:58.528244 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:58.528535 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/instance:
uuid: "346fb438dba74534836cebb489aa44d2"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.529456 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:58.529911 28452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.530052 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.530097 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root
uuid: "346fb438dba74534836cebb489aa44d2"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.530155 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:58.535147 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:58.535415 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:58.535558 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:58.535753 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:58.535781 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.535815 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:58.535842 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.538525 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.193:44235
I20240503 08:12:58.538563 28515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.193:44235 every 8 connection(s)
I20240503 08:12:58.538923 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:58.541476 28520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.542410 28516 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39047
I20240503 08:12:58.542482 28516 heartbeater.cc:464] Registering TS with master...
W20240503 08:12:58.542537 28523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.542685 28516 heartbeater.cc:511] Master 127.16.212.254:39047 requested a full tablet report, sending...
I20240503 08:12:58.542968 28387 ts_manager.cc:194] Registered new tserver with Master: 346fb438dba74534836cebb489aa44d2 (127.16.212.193:44235)
I20240503 08:12:58.543547 28387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43406
W20240503 08:12:58.543651 28521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.544719 17235 server_base.cc:1034] running on GCE node
I20240503 08:12:58.544831 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:58.544868 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:58.544886 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723978544886 us; error 0 us; skew 500 ppm
I20240503 08:12:58.545717 17235 webserver.cc:460] Webserver started at http://127.16.212.194:33221/ using document root <none> and password file <none>
I20240503 08:12:58.545814 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:58.545872 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:58.545930 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:58.546236 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/instance:
uuid: "a215981f3cf94161ba25daa9e8b3f7aa"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.547178 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:58.547654 28528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.547780 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.547823 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root
uuid: "a215981f3cf94161ba25daa9e8b3f7aa"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.547875 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:58.563373 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:58.563577 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:58.563755 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:58.563958 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:58.563984 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.564020 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:58.564050 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.566757 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.194:46319
I20240503 08:12:58.566799 28591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.194:46319 every 8 connection(s)
I20240503 08:12:58.567128 17235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 08:12:58.569561 28596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.570191 28592 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39047
I20240503 08:12:58.570256 28592 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:58.570402 28592 heartbeater.cc:511] Master 127.16.212.254:39047 requested a full tablet report, sending...
W20240503 08:12:58.570454 28595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.570689 28387 ts_manager.cc:194] Registered new tserver with Master: a215981f3cf94161ba25daa9e8b3f7aa (127.16.212.194:46319)
I20240503 08:12:58.571004 28387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43418
I20240503 08:12:58.571702 17235 server_base.cc:1034] running on GCE node
W20240503 08:12:58.572564 28598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 08:12:58.572714 17235 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 08:12:58.572750 17235 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 08:12:58.572773 17235 hybrid_clock.cc:648] HybridClock initialized: now 1714723978572773 us; error 0 us; skew 500 ppm
I20240503 08:12:58.573442 17235 webserver.cc:460] Webserver started at http://127.16.212.195:45517/ using document root <none> and password file <none>
I20240503 08:12:58.573526 17235 fs_manager.cc:359] Metadata directory not provided
I20240503 08:12:58.573575 17235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 08:12:58.573637 17235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 08:12:58.573908 17235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/instance:
uuid: "77d5740a6aa74347be19f5b66d7442f2"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.574779 17235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 08:12:58.575233 28603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.575368 17235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.575412 17235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root
uuid: "77d5740a6aa74347be19f5b66d7442f2"
format_stamp: "Formatted at 2024-05-03 08:12:58 on dist-test-slave-zl2t"
I20240503 08:12:58.575462 17235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 08:12:58.584133 17235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 08:12:58.584322 17235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 08:12:58.584467 17235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 08:12:58.584673 17235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 08:12:58.584702 17235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.584735 17235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 08:12:58.584764 17235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.587587 17235 rpc_server.cc:307] RPC server started. Bound to: 127.16.212.195:45433
I20240503 08:12:58.587653 28666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.212.195:45433 every 8 connection(s)
I20240503 08:12:58.590973 28667 heartbeater.cc:347] Connected to a master server at 127.16.212.254:39047
I20240503 08:12:58.591037 28667 heartbeater.cc:464] Registering TS with master...
I20240503 08:12:58.591150 28667 heartbeater.cc:511] Master 127.16.212.254:39047 requested a full tablet report, sending...
I20240503 08:12:58.591406 28387 ts_manager.cc:194] Registered new tserver with Master: 77d5740a6aa74347be19f5b66d7442f2 (127.16.212.195:45433)
I20240503 08:12:58.591714 28387 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43422
I20240503 08:12:58.592182 17235 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004403448s
I20240503 08:12:58.592527 17235 test_util.cc:268] Using random seed: 1235337487
I20240503 08:12:58.597640 28387 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43430:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 08:12:58.597772 28387 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 08:12:58.602388 28556 tablet_service.cc:1467] Processing CreateTablet for tablet ed79d2ecddc7447eadcdf1b581d723b3 (DEFAULT_TABLE table=test-workload [id=7cc31815c6c34786b4a42f0a59dc00f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.602388 28631 tablet_service.cc:1467] Processing CreateTablet for tablet ed79d2ecddc7447eadcdf1b581d723b3 (DEFAULT_TABLE table=test-workload [id=7cc31815c6c34786b4a42f0a59dc00f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.602562 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed79d2ecddc7447eadcdf1b581d723b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.602586 28631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed79d2ecddc7447eadcdf1b581d723b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.602685 28480 tablet_service.cc:1467] Processing CreateTablet for tablet ed79d2ecddc7447eadcdf1b581d723b3 (DEFAULT_TABLE table=test-workload [id=7cc31815c6c34786b4a42f0a59dc00f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 08:12:58.602784 28480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed79d2ecddc7447eadcdf1b581d723b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 08:12:58.603938 28682 tablet_bootstrap.cc:492] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Bootstrap starting.
I20240503 08:12:58.604174 28683 tablet_bootstrap.cc:492] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Bootstrap starting.
I20240503 08:12:58.604339 28682 tablet_bootstrap.cc:654] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.604444 28683 tablet_bootstrap.cc:654] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.604748 28684 tablet_bootstrap.cc:492] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Bootstrap starting.
I20240503 08:12:58.604857 28682 tablet_bootstrap.cc:492] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: No bootstrap required, opened a new log
I20240503 08:12:58.604908 28682 ts_tablet_manager.cc:1367] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:58.605095 28684 tablet_bootstrap.cc:654] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Neither blocks nor log segments found. Creating new log.
I20240503 08:12:58.605069 28682 raft_consensus.cc:348] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.605185 28682 raft_consensus.cc:374] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.605223 28682 raft_consensus.cc:729] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a215981f3cf94161ba25daa9e8b3f7aa, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.605316 28682 consensus_queue.cc:260] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.605504 28682 ts_tablet_manager.cc:1398] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:58.605551 28592 heartbeater.cc:503] Master 127.16.212.254:39047 was elected leader, sending a full tablet report...
I20240503 08:12:58.605578 28683 tablet_bootstrap.cc:492] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: No bootstrap required, opened a new log
I20240503 08:12:58.605641 28683 ts_tablet_manager.cc:1367] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:58.605803 28684 tablet_bootstrap.cc:492] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: No bootstrap required, opened a new log
I20240503 08:12:58.605857 28684 ts_tablet_manager.cc:1367] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240503 08:12:58.605815 28683 raft_consensus.cc:348] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.605891 28683 raft_consensus.cc:374] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.605906 28683 raft_consensus.cc:729] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346fb438dba74534836cebb489aa44d2, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.605942 28683 consensus_queue.cc:260] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.605993 28684 raft_consensus.cc:348] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.606046 28684 raft_consensus.cc:374] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 08:12:58.606065 28684 raft_consensus.cc:729] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d5740a6aa74347be19f5b66d7442f2, State: Initialized, Role: FOLLOWER
I20240503 08:12:58.606103 28683 ts_tablet_manager.cc:1398] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240503 08:12:58.606118 28684 consensus_queue.cc:260] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.606113 28516 heartbeater.cc:503] Master 127.16.212.254:39047 was elected leader, sending a full tablet report...
I20240503 08:12:58.606328 28667 heartbeater.cc:503] Master 127.16.212.254:39047 was elected leader, sending a full tablet report...
I20240503 08:12:58.606312 28684 ts_tablet_manager.cc:1398] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240503 08:12:58.913110 28689 raft_consensus.cc:482] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 08:12:58.913219 28689 raft_consensus.cc:504] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.913444 28689 leader_election.cc:288] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a215981f3cf94161ba25daa9e8b3f7aa (127.16.212.194:46319), 77d5740a6aa74347be19f5b66d7442f2 (127.16.212.195:45433)
I20240503 08:12:58.916069 28566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed79d2ecddc7447eadcdf1b581d723b3" candidate_uuid: "346fb438dba74534836cebb489aa44d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" is_pre_election: true
I20240503 08:12:58.916069 28641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed79d2ecddc7447eadcdf1b581d723b3" candidate_uuid: "346fb438dba74534836cebb489aa44d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d5740a6aa74347be19f5b66d7442f2" is_pre_election: true
I20240503 08:12:58.916215 28566 raft_consensus.cc:2454] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346fb438dba74534836cebb489aa44d2 in term 0.
I20240503 08:12:58.916219 28641 raft_consensus.cc:2454] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346fb438dba74534836cebb489aa44d2 in term 0.
I20240503 08:12:58.916364 28455 leader_election.cc:302] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346fb438dba74534836cebb489aa44d2, 77d5740a6aa74347be19f5b66d7442f2; no voters:
I20240503 08:12:58.916472 28689 raft_consensus.cc:2781] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 08:12:58.916514 28689 raft_consensus.cc:482] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 08:12:58.916534 28689 raft_consensus.cc:3037] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.917171 28689 raft_consensus.cc:504] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.917276 28689 leader_election.cc:288] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [CANDIDATE]: Term 1 election: Requested vote from peers a215981f3cf94161ba25daa9e8b3f7aa (127.16.212.194:46319), 77d5740a6aa74347be19f5b66d7442f2 (127.16.212.195:45433)
I20240503 08:12:58.917482 28566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed79d2ecddc7447eadcdf1b581d723b3" candidate_uuid: "346fb438dba74534836cebb489aa44d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a215981f3cf94161ba25daa9e8b3f7aa"
I20240503 08:12:58.917528 28641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed79d2ecddc7447eadcdf1b581d723b3" candidate_uuid: "346fb438dba74534836cebb489aa44d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d5740a6aa74347be19f5b66d7442f2"
I20240503 08:12:58.917577 28641 raft_consensus.cc:3037] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.917565 28566 raft_consensus.cc:3037] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 08:12:58.918294 28566 raft_consensus.cc:2454] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346fb438dba74534836cebb489aa44d2 in term 1.
I20240503 08:12:58.918294 28641 raft_consensus.cc:2454] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346fb438dba74534836cebb489aa44d2 in term 1.
I20240503 08:12:58.918535 28455 leader_election.cc:302] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346fb438dba74534836cebb489aa44d2, 77d5740a6aa74347be19f5b66d7442f2; no voters:
I20240503 08:12:58.918644 28689 raft_consensus.cc:2781] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 08:12:58.918782 28689 raft_consensus.cc:686] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [term 1 LEADER]: Becoming Leader. State: Replica: 346fb438dba74534836cebb489aa44d2, State: Running, Role: LEADER
I20240503 08:12:58.918843 28689 consensus_queue.cc:237] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } }
I20240503 08:12:58.919405 28387 catalog_manager.cc:5484] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346fb438dba74534836cebb489aa44d2 (127.16.212.193). New cstate: current_term: 1 leader_uuid: "346fb438dba74534836cebb489aa44d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "346fb438dba74534836cebb489aa44d2" member_type: VOTER last_known_addr { host: "127.16.212.193" port: 44235 } health_report { overall_health: HEALTHY } } }
I20240503 08:12:58.962694 28566 raft_consensus.cc:1261] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa [term 1 FOLLOWER]: Refusing update from remote peer 346fb438dba74534836cebb489aa44d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:58.962694 28641 raft_consensus.cc:1261] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2 [term 1 FOLLOWER]: Refusing update from remote peer 346fb438dba74534836cebb489aa44d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 08:12:58.962973 28689 consensus_queue.cc:1035] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a215981f3cf94161ba25daa9e8b3f7aa" member_type: VOTER last_known_addr { host: "127.16.212.194" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:58.963115 28692 consensus_queue.cc:1035] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77d5740a6aa74347be19f5b66d7442f2" member_type: VOTER last_known_addr { host: "127.16.212.195" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 08:12:59.094601 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.094887 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.096061 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.116442 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.022s user 0.017s sys 0.000s Metrics: {"bytes_written":277752,"cfile_init":1,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650,"thread_start_us":69,"threads_started":1}
I20240503 08:12:59.116755 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89132 bytes on disk
I20240503 08:12:59.116909 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.119851 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.021s user 0.006s sys 0.010s Metrics: {"bytes_written":277752,"cfile_init":1,"dirs.queue_time_us":1661,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1100,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650,"thread_start_us":55,"threads_started":1}
I20240503 08:12:59.120190 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89132 bytes on disk
I20240503 08:12:59.120378 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.120412 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.025s user 0.010s sys 0.004s Metrics: {"bytes_written":277752,"cfile_init":1,"dirs.queue_time_us":3400,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":1699,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650,"thread_start_us":54,"threads_started":1}
I20240503 08:12:59.120669 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89132 bytes on disk
I20240503 08:12:59.120868 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.252369 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.252676 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.258680 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.264665 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":275016,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":862,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1149,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:59.265231 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014720
I20240503 08:12:59.264627 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":273745,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":630,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":748,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:12:59.269199 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014720
I20240503 08:12:59.294517 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.036s user 0.014s sys 0.000s Metrics: {"bytes_written":275016,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":743,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:12:59.294718 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88472 bytes on disk
I20240503 08:12:59.300276 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.300531 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014720
I20240503 08:12:59.310593 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.037s user 0.024s sys 0.004s Metrics: {"bytes_written":547491,"cfile_cache_hit":10,"cfile_cache_hit_bytes":360848,"cfile_cache_miss":42,"cfile_cache_miss_bytes":995776,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":229,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":836,"drs_written":1,"lbm_read_time_us":807,"lbm_reads_lt_1ms":66,"lbm_write_time_us":3948,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":42,"num_input_rowsets":2,"peak_mem_usage":286919,"rows_written":19150,"thread_start_us":588,"threads_started":5}
I20240503 08:12:59.311355 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 182393 bytes on disk
I20240503 08:12:59.311548 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.313498 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.047s user 0.026s sys 0.004s Metrics: {"bytes_written":548856,"cfile_cache_hit":10,"cfile_cache_hit_bytes":361813,"cfile_cache_miss":42,"cfile_cache_miss_bytes":998511,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":277,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":1310,"drs_written":1,"lbm_read_time_us":885,"lbm_reads_lt_1ms":66,"lbm_write_time_us":1309,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":287679,"rows_written":19200,"thread_start_us":279,"threads_started":5}
I20240503 08:12:59.313795 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 182871 bytes on disk
I20240503 08:12:59.314064 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":30,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.335865 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.035s user 0.021s sys 0.007s Metrics: {"bytes_written":548856,"cfile_cache_hit":11,"cfile_cache_hit_bytes":391653,"cfile_cache_miss":41,"cfile_cache_miss_bytes":968671,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":549,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":920,"drs_written":1,"lbm_read_time_us":637,"lbm_reads_lt_1ms":61,"lbm_write_time_us":1199,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":287679,"rows_written":19200,"thread_start_us":286,"threads_started":5}
I20240503 08:12:59.336207 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 182871 bytes on disk
I20240503 08:12:59.336495 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.418118 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.424783 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.425871 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.439296 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.021s user 0.014s sys 0.000s Metrics: {"bytes_written":272432,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":970,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:12:59.440080 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 87033 bytes on disk
I20240503 08:12:59.440276 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.440490 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014662
I20240503 08:12:59.439296 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":284300,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":832,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20240503 08:12:59.442301 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 91262 bytes on disk
I20240503 08:12:59.443593 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.443974 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014656
I20240503 08:12:59.446966 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.022s user 0.013s sys 0.000s Metrics: {"bytes_written":283121,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":731,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1018,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20240503 08:12:59.447142 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014657
I20240503 08:12:59.490991 28376 maintenance_manager.cc:419] P a325bde86de4459292246df81b755452: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.003577
I20240503 08:12:59.494292 28375 maintenance_manager.cc:643] P a325bde86de4459292246df81b755452: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":6635,"cfile_init":1,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":168,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":5,"thread_start_us":57,"threads_started":1}
I20240503 08:12:59.494753 28376 maintenance_manager.cc:419] P a325bde86de4459292246df81b755452: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 487 bytes on disk
I20240503 08:12:59.494951 28375 maintenance_manager.cc:643] P a325bde86de4459292246df81b755452: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.501267 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.057s user 0.040s sys 0.000s Metrics: {"bytes_written":829918,"cfile_cache_hit":12,"cfile_cache_hit_bytes":564290,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1500430,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":823,"drs_written":1,"lbm_read_time_us":1250,"lbm_reads_lt_1ms":70,"lbm_write_time_us":2032,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":438641,"rows_written":29100}
I20240503 08:12:59.501538 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 282747 bytes on disk
I20240503 08:12:59.501708 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.508445 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.068s user 0.034s sys 0.004s Metrics: {"bytes_written":814673,"cfile_cache_hit":12,"cfile_cache_hit_bytes":555630,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1476894,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":693,"drs_written":1,"lbm_read_time_us":1168,"lbm_reads_lt_1ms":70,"lbm_write_time_us":1651,"lbm_writes_lt_1ms":54,"num_input_rowsets":2,"peak_mem_usage":431776,"rows_written":28650}
I20240503 08:12:59.508630 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 278200 bytes on disk
I20240503 08:12:59.508929 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.517092 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.070s user 0.028s sys 0.008s Metrics: {"bytes_written":823202,"cfile_cache_hit":12,"cfile_cache_hit_bytes":562916,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1494890,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":1324,"drs_written":1,"lbm_read_time_us":1161,"lbm_reads_lt_1ms":74,"lbm_write_time_us":2971,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":53,"num_input_rowsets":2,"peak_mem_usage":437111,"rows_written":29000}
I20240503 08:12:59.517258 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 281835 bytes on disk
I20240503 08:12:59.517382 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.610706 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.613866 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.618978 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.630901 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.017s user 0.012s sys 0.004s Metrics: {"bytes_written":277800,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":977,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:59.631110 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1061709 bytes of WAL
I20240503 08:12:59.631186 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:12:59.631223 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000001 (ops 1-586)
I20240503 08:12:59.631644 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:59.631758 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014477
I20240503 08:12:59.642125 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.031s user 0.015s sys 0.000s Metrics: {"bytes_written":284292,"cfile_init":1,"dirs.queue_time_us":4623,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1125,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20240503 08:12:59.642310 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1063040 bytes of WAL
I20240503 08:12:59.645205 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:12:59.645268 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000001 (ops 1-641)
I20240503 08:12:59.645722 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:59.645843 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 92835 bytes on disk
I20240503 08:12:59.645956 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.646075 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014481
I20240503 08:12:59.650302 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.031s user 0.012s sys 0.004s Metrics: {"bytes_written":281690,"cfile_init":1,"dirs.queue_time_us":5895,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":1125,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":986,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:12:59.650564 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1061552 bytes of WAL
I20240503 08:12:59.650699 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:12:59.650763 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000001 (ops 1-641)
I20240503 08:12:59.651198 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:12:59.651319 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014478
I20240503 08:12:59.711709 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.060s user 0.049s sys 0.001s Metrics: {"bytes_written":1099848,"cfile_cache_hit":12,"cfile_cache_hit_bytes":616514,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2149636,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":943,"drs_written":1,"lbm_read_time_us":1565,"lbm_reads_lt_1ms":86,"lbm_write_time_us":4000,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":586541,"rows_written":38800}
I20240503 08:12:59.712044 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 385182 bytes on disk
I20240503 08:12:59.712255 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.719386 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.073s user 0.047s sys 0.000s Metrics: {"bytes_written":1093716,"cfile_cache_hit":12,"cfile_cache_hit_bytes":616945,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2132009,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":92,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":902,"drs_written":1,"lbm_read_time_us":1607,"lbm_reads_lt_1ms":82,"lbm_write_time_us":2360,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":582731,"rows_written":38550,"spinlock_wait_cycles":384}
I20240503 08:12:59.719571 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 382468 bytes on disk
I20240503 08:12:59.719710 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.720424 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.089s user 0.047s sys 0.003s Metrics: {"bytes_written":1098775,"cfile_cache_hit":12,"cfile_cache_hit_bytes":614345,"cfile_cache_miss":62,"cfile_cache_miss_bytes":2148267,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":481,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1000,"drs_written":1,"lbm_read_time_us":1587,"lbm_reads_lt_1ms":86,"lbm_write_time_us":2633,"lbm_writes_lt_1ms":64,"mutex_wait_us":23,"num_input_rowsets":2,"peak_mem_usage":585776,"rows_written":38750}
I20240503 08:12:59.720579 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 384663 bytes on disk
I20240503 08:12:59.720850 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.810909 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.812386 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.813652 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:12:59.822757 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":285563,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":808,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":673,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9950}
I20240503 08:12:59.822921 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014456
I20240503 08:12:59.828545 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":284399,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":823,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20240503 08:12:59.828737 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 91327 bytes on disk
I20240503 08:12:59.828876 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.828971 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014455
I20240503 08:12:59.860066 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.046s user 0.012s sys 0.002s Metrics: {"bytes_written":277768,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":759,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":808,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:12:59.860260 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88813 bytes on disk
I20240503 08:12:59.863627 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.863787 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014457
I20240503 08:12:59.923643 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.095s user 0.053s sys 0.004s Metrics: {"bytes_written":1376205,"cfile_cache_hit":11,"cfile_cache_hit_bytes":560669,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2917187,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":790,"drs_written":1,"lbm_read_time_us":2138,"lbm_reads_lt_1ms":91,"lbm_write_time_us":2840,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":736738,"rows_written":48650}
I20240503 08:12:59.923852 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 493332 bytes on disk
I20240503 08:12:59.924010 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.923245 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.100s user 0.041s sys 0.015s Metrics: {"bytes_written":1372143,"cfile_cache_hit":11,"cfile_cache_hit_bytes":561227,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2905377,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":3419,"dirs.run_cpu_time_us":458,"dirs.run_wall_time_us":4339,"drs_written":1,"lbm_read_time_us":1507,"lbm_reads_lt_1ms":95,"lbm_write_time_us":6077,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":75,"num_input_rowsets":2,"peak_mem_usage":734448,"rows_written":48500,"spinlock_wait_cycles":384}
I20240503 08:12:59.924793 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 491742 bytes on disk
I20240503 08:12:59.924922 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:12:59.965725 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.102s user 0.062s sys 0.000s Metrics: {"bytes_written":1370789,"cfile_cache_hit":12,"cfile_cache_hit_bytes":667088,"cfile_cache_miss":70,"cfile_cache_miss_bytes":2795658,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":260,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1317,"drs_written":1,"lbm_read_time_us":1687,"lbm_reads_lt_1ms":90,"lbm_write_time_us":7849,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":75,"num_input_rowsets":2,"peak_mem_usage":733683,"rows_written":48450,"spinlock_wait_cycles":384}
I20240503 08:12:59.965950 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 491045 bytes on disk
I20240503 08:12:59.966277 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.016878 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.017009 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.017002 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.031848 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":272581,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":943,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:13:00.031852 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.015s user 0.011s sys 0.000s Metrics: {"bytes_written":276410,"cfile_init":1,"dirs.queue_time_us":121,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1139,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":738,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:13:00.032058 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014122
I20240503 08:13:00.032126 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88445 bytes on disk
I20240503 08:13:00.032255 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.032366 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014123
I20240503 08:13:00.047420 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.024s user 0.014s sys 0.001s Metrics: {"bytes_written":277826,"cfile_init":1,"dirs.queue_time_us":1372,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":991,"drs_written":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4,"lbm_write_time_us":789,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:00.047647 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88947 bytes on disk
I20240503 08:13:00.047825 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.049597 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014123
I20240503 08:13:00.138347 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.106s user 0.058s sys 0.010s Metrics: {"bytes_written":1638516,"cfile_cache_hit":12,"cfile_cache_hit_bytes":718120,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3449464,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":858,"drs_written":1,"lbm_read_time_us":3100,"lbm_reads_lt_1ms":102,"lbm_write_time_us":5592,"lbm_writes_lt_1ms":86,"num_input_rowsets":2,"peak_mem_usage":880828,"rows_written":58100}
I20240503 08:13:00.139463 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 595875 bytes on disk
I20240503 08:13:00.141863 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.163491 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.114s user 0.059s sys 0.012s Metrics: {"bytes_written":1638516,"cfile_cache_hit":12,"cfile_cache_hit_bytes":718625,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3448695,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1800,"drs_written":1,"lbm_read_time_us":3520,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":101,"lbm_write_time_us":6955,"lbm_writes_lt_1ms":86,"num_input_rowsets":2,"peak_mem_usage":880823,"rows_written":58100}
I20240503 08:13:00.163691 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 595875 bytes on disk
I20240503 08:13:00.163810 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.181340 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.149s user 0.064s sys 0.004s Metrics: {"bytes_written":1638516,"cfile_cache_hit":12,"cfile_cache_hit_bytes":608202,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3559996,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":932,"drs_written":1,"lbm_read_time_us":10498,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":104,"lbm_write_time_us":6529,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":84,"num_input_rowsets":2,"peak_mem_usage":880823,"rows_written":58100,"spinlock_wait_cycles":1920}
I20240503 08:13:00.181733 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 595875 bytes on disk
I20240503 08:13:00.182511 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.223197 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.223498 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.224589 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.249570 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.026s user 0.012s sys 0.005s Metrics: {"bytes_written":280188,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":724,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2193,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:13:00.249806 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1051482 bytes of WAL
I20240503 08:13:00.249895 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.025s user 0.012s sys 0.004s Metrics: {"bytes_written":280188,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":917,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":718,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:13:00.250036 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1048229 bytes of WAL
I20240503 08:13:00.250135 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:00.250262 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000002 (ops 642-1272)
I20240503 08:13:00.250494 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.027s user 0.010s sys 0.003s Metrics: {"bytes_written":279224,"cfile_init":1,"dirs.queue_time_us":116,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":854,"drs_written":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5735,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:13:00.250641 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:00.250677 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000002 (ops 587-1165)
I20240503 08:13:00.250705 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1050146 bytes of WAL
I20240503 08:13:00.250850 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:00.250939 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89879 bytes on disk
I20240503 08:13:00.251050 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.251049 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:00.251178 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89879 bytes on disk
I20240503 08:13:00.251181 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014242
I20240503 08:13:00.251340 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.251439 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014242
I20240503 08:13:00.253031 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:00.253093 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000002 (ops 642-1268)
I20240503 08:13:00.253422 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:00.253517 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.014243
I20240503 08:13:00.346307 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.095s user 0.066s sys 0.011s Metrics: {"bytes_written":1914545,"cfile_cache_hit":13,"cfile_cache_hit_bytes":801749,"cfile_cache_miss":91,"cfile_cache_miss_bytes":4076085,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":831,"drs_written":1,"lbm_read_time_us":2723,"lbm_reads_lt_1ms":111,"lbm_write_time_us":4074,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1029495,"rows_written":67850,"spinlock_wait_cycles":896}
I20240503 08:13:00.346643 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 701848 bytes on disk
I20240503 08:13:00.346824 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.383760 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.132s user 0.074s sys 0.007s Metrics: {"bytes_written":1914545,"cfile_cache_hit":13,"cfile_cache_hit_bytes":801749,"cfile_cache_miss":91,"cfile_cache_miss_bytes":4076085,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":433,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":811,"drs_written":1,"lbm_read_time_us":3421,"lbm_reads_lt_1ms":111,"lbm_write_time_us":4093,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1029495,"rows_written":67850}
I20240503 08:13:00.384115 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 701848 bytes on disk
I20240503 08:13:00.384332 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.399502 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.146s user 0.073s sys 0.002s Metrics: {"bytes_written":1913142,"cfile_cache_hit":13,"cfile_cache_hit_bytes":800849,"cfile_cache_miss":91,"cfile_cache_miss_bytes":4074185,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":222,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":970,"drs_written":1,"lbm_read_time_us":8602,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":113,"lbm_write_time_us":9914,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":96,"num_input_rowsets":2,"peak_mem_usage":1028735,"rows_written":67800}
I20240503 08:13:00.399889 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 701146 bytes on disk
I20240503 08:13:00.400185 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.440829 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.445483 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.448410 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.454764 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":274960,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:13:00.455011 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 87995 bytes on disk
I20240503 08:13:00.455138 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.455232 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013996
I20240503 08:13:00.486037 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.040s user 0.012s sys 0.004s Metrics: {"bytes_written":277724,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":880,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1398,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:00.486346 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013995
I20240503 08:13:00.509389 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.061s user 0.016s sys 0.000s Metrics: {"bytes_written":277724,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":942,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:00.510597 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88933 bytes on disk
I20240503 08:13:00.511035 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":215,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.511322 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013995
I20240503 08:13:00.548835 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.094s user 0.076s sys 0.004s Metrics: {"bytes_written":2180900,"cfile_cache_hit":10,"cfile_cache_hit_bytes":443004,"cfile_cache_miss":108,"cfile_cache_miss_bytes":5130936,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":1263,"drs_written":1,"lbm_read_time_us":2969,"lbm_reads_lt_1ms":128,"lbm_write_time_us":4704,"lbm_writes_lt_1ms":111,"num_input_rowsets":2,"peak_mem_usage":1174345,"rows_written":77350,"spinlock_wait_cycles":384}
I20240503 08:13:00.549090 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 805687 bytes on disk
I20240503 08:13:00.549232 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.595604 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.109s user 0.065s sys 0.020s Metrics: {"bytes_written":2184848,"cfile_cache_hit":11,"cfile_cache_hit_bytes":626522,"cfile_cache_miss":107,"cfile_cache_miss_bytes":4959032,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":827,"drs_written":1,"lbm_read_time_us":3316,"lbm_reads_lt_1ms":131,"lbm_write_time_us":6969,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":110,"num_input_rowsets":2,"peak_mem_usage":1176661,"rows_written":77500,"spinlock_wait_cycles":384}
I20240503 08:13:00.595861 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 807503 bytes on disk
I20240503 08:13:00.596060 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.612476 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.101s user 0.060s sys 0.015s Metrics: {"bytes_written":2184848,"cfile_cache_hit":11,"cfile_cache_hit_bytes":626522,"cfile_cache_miss":107,"cfile_cache_miss_bytes":4959032,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":245,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":647,"drs_written":1,"lbm_read_time_us":3392,"lbm_reads_lt_1ms":127,"lbm_write_time_us":4620,"lbm_writes_lt_1ms":111,"num_input_rowsets":2,"peak_mem_usage":1176661,"rows_written":77500}
I20240503 08:13:00.612731 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 807503 bytes on disk
I20240503 08:13:00.612920 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.691220 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.694058 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.706429 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":273704,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":1076,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":629,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20240503 08:13:00.706858 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013716
I20240503 08:13:00.708005 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.721729 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.028s user 0.013s sys 0.000s Metrics: {"bytes_written":274990,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":787,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":889,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:13:00.722538 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013714
I20240503 08:13:00.731261 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.023s user 0.010s sys 0.004s Metrics: {"bytes_written":280313,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":1081,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1036,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:13:00.731433 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 91106 bytes on disk
I20240503 08:13:00.731562 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.731652 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013773
I20240503 08:13:00.874434 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.150s user 0.094s sys 0.004s Metrics: {"bytes_written":2447748,"cfile_cache_hit":11,"cfile_cache_hit_bytes":472867,"cfile_cache_miss":119,"cfile_cache_miss_bytes":5810451,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":837,"drs_written":1,"lbm_read_time_us":6189,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":142,"lbm_write_time_us":5186,"lbm_writes_lt_1ms":122,"num_input_rowsets":2,"peak_mem_usage":1331936,"rows_written":87050,"spinlock_wait_cycles":768}
I20240503 08:13:00.874809 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 914338 bytes on disk
I20240503 08:13:00.874974 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.895445 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.164s user 0.092s sys 0.006s Metrics: {"bytes_written":2456837,"cfile_cache_hit":11,"cfile_cache_hit_bytes":477363,"cfile_cache_miss":119,"cfile_cache_miss_bytes":5820215,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":862,"drs_written":1,"lbm_read_time_us":4554,"lbm_reads_lt_1ms":139,"lbm_write_time_us":8762,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":122,"num_input_rowsets":2,"peak_mem_usage":1334986,"rows_written":87250}
I20240503 08:13:00.895624 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 916940 bytes on disk
I20240503 08:13:00.895646 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.895748 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:00.906013 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.923079 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.027s user 0.010s sys 0.004s Metrics: {"bytes_written":272488,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":904,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1267,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:13:00.923316 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013943
I20240503 08:13:00.952134 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.046s user 0.016s sys 0.001s Metrics: {"bytes_written":279129,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":1095,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2711,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:13:00.952324 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1072234 bytes of WAL
I20240503 08:13:00.952396 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:00.952426 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000003 (ops 1273-1913)
I20240503 08:13:00.952845 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:00.952948 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013883
I20240503 08:13:00.954443 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.247s user 0.080s sys 0.020s Metrics: {"bytes_written":2442947,"cfile_cache_hit":13,"cfile_cache_hit_bytes":832347,"cfile_cache_miss":117,"cfile_cache_miss_bytes":5435939,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":881,"drs_written":1,"lbm_read_time_us":8959,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":140,"lbm_write_time_us":17402,"lbm_writes_1-10_ms":6,"lbm_writes_lt_1ms":115,"num_input_rowsets":2,"peak_mem_usage":1328736,"rows_written":86850}
I20240503 08:13:00.954769 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:00.999773 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.044s user 0.014s sys 0.002s Metrics: {"bytes_written":366618,"cfile_init":1,"dirs.queue_time_us":87,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1595,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":839,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12700,"spinlock_wait_cycles":384}
I20240503 08:13:00.999992 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1058432 bytes of WAL
I20240503 08:13:01.000080 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:01.000110 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000003 (ops 1269-1895)
I20240503 08:13:01.000474 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:01.001080 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013913
I20240503 08:13:01.063807 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.140s user 0.085s sys 0.019s Metrics: {"bytes_written":2716034,"cfile_cache_hit":12,"cfile_cache_hit_bytes":649438,"cfile_cache_miss":130,"cfile_cache_miss_bytes":6323158,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":356,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1017,"drs_written":1,"lbm_read_time_us":3155,"lbm_reads_lt_1ms":154,"lbm_write_time_us":7031,"lbm_writes_lt_1ms":132,"num_input_rowsets":2,"peak_mem_usage":1485602,"rows_written":96500,"spinlock_wait_cycles":1536}
I20240503 08:13:01.064005 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1055251 bytes of WAL
I20240503 08:13:01.064083 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:01.064114 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000003 (ops 1166-1746)
I20240503 08:13:01.064538 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:01.064639 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1022146 bytes on disk
I20240503 08:13:01.064752 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.105374 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.152s user 0.091s sys 0.011s Metrics: {"bytes_written":2727646,"cfile_cache_hit":11,"cfile_cache_hit_bytes":624614,"cfile_cache_miss":131,"cfile_cache_miss_bytes":6381186,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":778,"drs_written":1,"lbm_read_time_us":9065,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":154,"lbm_write_time_us":5360,"lbm_writes_lt_1ms":133,"num_input_rowsets":2,"peak_mem_usage":1492651,"rows_written":96950,"spinlock_wait_cycles":384}
I20240503 08:13:01.106423 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1027474 bytes on disk
I20240503 08:13:01.106642 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.115955 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.127002 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.134227 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":281664,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":765,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":675,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:13:01.134658 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 90403 bytes on disk
I20240503 08:13:01.134878 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.135046 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013810
I20240503 08:13:01.143883 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":275099,"cfile_init":1,"dirs.queue_time_us":96,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1202,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":720,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:13:01.144670 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013808
I20240503 08:13:01.165221 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.164s user 0.096s sys 0.009s Metrics: {"bytes_written":2800078,"cfile_cache_hit":11,"cfile_cache_hit_bytes":543027,"cfile_cache_miss":131,"cfile_cache_miss_bytes":6650163,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":758,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1105,"drs_written":1,"lbm_read_time_us":4061,"lbm_reads_lt_1ms":159,"lbm_write_time_us":5783,"lbm_writes_lt_1ms":137,"num_input_rowsets":2,"peak_mem_usage":1531895,"rows_written":99550}
I20240503 08:13:01.165500 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1057091 bytes on disk
I20240503 08:13:01.165791 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.196421 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.240415 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.043s user 0.014s sys 0.000s Metrics: {"bytes_written":277814,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":952,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2257,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:01.241722 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013779
I20240503 08:13:01.306217 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.171s user 0.102s sys 0.010s Metrics: {"bytes_written":2988940,"cfile_cache_hit":11,"cfile_cache_hit_bytes":625328,"cfile_cache_miss":139,"cfile_cache_miss_bytes":7074586,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":808,"drs_written":1,"lbm_read_time_us":6757,"lbm_reads_lt_1ms":159,"lbm_write_time_us":11723,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":142,"num_input_rowsets":2,"peak_mem_usage":1644477,"rows_written":106300}
I20240503 08:13:01.306418 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1133402 bytes on disk
I20240503 08:13:01.307194 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.316782 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.172s user 0.112s sys 0.006s Metrics: {"bytes_written":2994353,"cfile_cache_hit":10,"cfile_cache_hit_bytes":443273,"cfile_cache_miss":142,"cfile_cache_miss_bytes":7272367,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":894,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":753,"drs_written":1,"lbm_read_time_us":6860,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":164,"lbm_write_time_us":9083,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":143,"num_input_rowsets":2,"peak_mem_usage":1647977,"rows_written":106500}
I20240503 08:13:01.317162 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1135876 bytes on disk
I20240503 08:13:01.317282 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.347733 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.347900 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.364818 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.017s user 0.010s sys 0.000s Metrics: {"bytes_written":272528,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":934,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:13:01.365211 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 87182 bytes on disk
I20240503 08:13:01.365506 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.365705 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013594
I20240503 08:13:01.370626 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":280303,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":706,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":674,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:13:01.370952 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 90079 bytes on disk
I20240503 08:13:01.371104 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.371645 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013594
I20240503 08:13:01.439599 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.198s user 0.103s sys 0.014s Metrics: {"bytes_written":3068123,"cfile_cache_hit":11,"cfile_cache_hit_bytes":621899,"cfile_cache_miss":145,"cfile_cache_miss_bytes":7288993,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":908,"drs_written":1,"lbm_read_time_us":7645,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":168,"lbm_write_time_us":8523,"lbm_writes_lt_1ms":146,"num_input_rowsets":2,"peak_mem_usage":1691745,"rows_written":109200}
I20240503 08:13:01.439796 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1165738 bytes on disk
I20240503 08:13:01.439919 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.440021 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.477283 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.037s user 0.015s sys 0.000s Metrics: {"bytes_written":308154,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":790,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4630,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":10650}
I20240503 08:13:01.477452 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013539
I20240503 08:13:01.584178 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.212s user 0.117s sys 0.014s Metrics: {"bytes_written":3257524,"cfile_cache_hit":13,"cfile_cache_hit_bytes":696332,"cfile_cache_miss":149,"cfile_cache_miss_bytes":7719102,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":989,"drs_written":1,"lbm_read_time_us":6687,"lbm_reads_lt_1ms":169,"lbm_write_time_us":12011,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":153,"num_input_rowsets":2,"peak_mem_usage":1802952,"rows_written":116050,"spinlock_wait_cycles":1664}
I20240503 08:13:01.585846 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.621609 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.035s user 0.013s sys 0.004s Metrics: {"bytes_written":313623,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":795,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1117,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10850}
I20240503 08:13:01.622303 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1051428 bytes of WAL
I20240503 08:13:01.624006 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:01.624145 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000004 (ops 1747-2325)
I20240503 08:13:01.625072 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:01.625218 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1343563 bytes on disk
I20240503 08:13:01.626596 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":37,"lbm_reads_lt_1ms":8}
I20240503 08:13:01.626785 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013424
I20240503 08:13:01.654187 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.288s user 0.120s sys 0.009s Metrics: {"bytes_written":3254285,"cfile_cache_hit":13,"cfile_cache_hit_bytes":756994,"cfile_cache_miss":149,"cfile_cache_miss_bytes":7652860,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2137,"drs_written":1,"lbm_read_time_us":9624,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":167,"lbm_write_time_us":16206,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":151,"num_input_rowsets":2,"peak_mem_usage":1801627,"rows_written":115950,"spinlock_wait_cycles":768}
I20240503 08:13:01.654796 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.713227 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.236s user 0.113s sys 0.016s Metrics: {"bytes_written":3367269,"cfile_cache_hit":14,"cfile_cache_hit_bytes":907660,"cfile_cache_miss":150,"cfile_cache_miss_bytes":7786698,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":1183,"drs_written":1,"lbm_read_time_us":11347,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":172,"lbm_write_time_us":12166,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":158,"num_input_rowsets":2,"peak_mem_usage":1863801,"rows_written":119850,"spinlock_wait_cycles":1408}
I20240503 08:13:01.713423 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.714668 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.060s user 0.022s sys 0.000s Metrics: {"bytes_written":396915,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":791,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1298,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":13700}
I20240503 08:13:01.714841 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1066170 bytes of WAL
I20240503 08:13:01.716389 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:01.716728 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000004 (ops 1914-2559)
I20240503 08:13:01.717443 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:01.717797 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013443
I20240503 08:13:01.754680 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.041s user 0.020s sys 0.000s Metrics: {"bytes_written":367622,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":1032,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2860,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12750}
I20240503 08:13:01.754923 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1055241 bytes of WAL
I20240503 08:13:01.755596 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:01.755743 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000004 (ops 1896-2522)
I20240503 08:13:01.756137 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:01.756237 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013352
I20240503 08:13:01.892755 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.266s user 0.129s sys 0.008s Metrics: {"bytes_written":3562795,"cfile_cache_hit":14,"cfile_cache_hit_bytes":947879,"cfile_cache_miss":160,"cfile_cache_miss_bytes":8262109,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":939,"drs_written":1,"lbm_read_time_us":5863,"lbm_reads_lt_1ms":180,"lbm_write_time_us":10707,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":167,"num_input_rowsets":2,"peak_mem_usage":1978152,"rows_written":126900,"spinlock_wait_cycles":384}
I20240503 08:13:01.892930 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1364370 bytes on disk
I20240503 08:13:01.893095 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.893185 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.923345 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.030s user 0.014s sys 0.006s Metrics: {"bytes_written":377065,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":840,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1547,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13100}
I20240503 08:13:01.923723 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013473
I20240503 08:13:01.948093 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.230s user 0.126s sys 0.014s Metrics: {"bytes_written":3636373,"cfile_cache_hit":12,"cfile_cache_hit_bytes":740197,"cfile_cache_miss":166,"cfile_cache_miss_bytes":8667613,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2664,"drs_written":1,"lbm_read_time_us":10248,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":192,"lbm_write_time_us":14487,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":169,"num_input_rowsets":2,"peak_mem_usage":2019983,"rows_written":129650,"spinlock_wait_cycles":2176}
I20240503 08:13:01.948604 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.960839 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.204s user 0.125s sys 0.015s Metrics: {"bytes_written":3720523,"cfile_cache_hit":11,"cfile_cache_hit_bytes":544114,"cfile_cache_miss":169,"cfile_cache_miss_bytes":9086696,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":971,"drs_written":1,"lbm_read_time_us":7035,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":196,"lbm_write_time_us":9383,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":175,"num_input_rowsets":2,"peak_mem_usage":2068873,"rows_written":132600,"spinlock_wait_cycles":384}
I20240503 08:13:01.961066 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1430668 bytes on disk
I20240503 08:13:01.961230 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:01.971576 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:01.975888 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.027s user 0.013s sys 0.003s Metrics: {"bytes_written":335857,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":689,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1117,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11700}
I20240503 08:13:01.976420 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013458
I20240503 08:13:01.996706 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":281569,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":907,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1507,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:13:01.996932 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013445
I20240503 08:13:02.131870 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.208s user 0.123s sys 0.023s Metrics: {"bytes_written":3925675,"cfile_cache_hit":12,"cfile_cache_hit_bytes":715126,"cfile_cache_miss":178,"cfile_cache_miss_bytes":9454500,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":905,"drs_written":1,"lbm_read_time_us":7500,"lbm_reads_lt_1ms":202,"lbm_write_time_us":13002,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":183,"num_input_rowsets":2,"peak_mem_usage":2188763,"rows_written":140000,"spinlock_wait_cycles":37504}
I20240503 08:13:02.132707 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1515995 bytes on disk
I20240503 08:13:02.133013 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.133999 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.157487 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":275018,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":1184,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:13:02.157792 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1057214 bytes of WAL
I20240503 08:13:02.157963 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:02.158133 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000005 (ops 2326-2907)
I20240503 08:13:02.159581 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:02.159840 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88040 bytes on disk
I20240503 08:13:02.160028 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.160182 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013410
I20240503 08:13:02.181610 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.205s user 0.143s sys 0.005s Metrics: {"bytes_written":3963317,"cfile_cache_hit":11,"cfile_cache_hit_bytes":655154,"cfile_cache_miss":181,"cfile_cache_miss_bytes":9614184,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":988,"drs_written":1,"lbm_read_time_us":5040,"lbm_reads_lt_1ms":209,"lbm_write_time_us":11007,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":184,"num_input_rowsets":2,"peak_mem_usage":2212098,"rows_written":141350,"spinlock_wait_cycles":1152,"thread_start_us":57,"threads_started":1}
I20240503 08:13:02.182157 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.234861 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.053s user 0.015s sys 0.000s Metrics: {"bytes_written":286906,"cfile_init":1,"dirs.queue_time_us":659,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":1716,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":695,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20240503 08:13:02.235232 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013391
I20240503 08:13:02.297633 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.301s user 0.136s sys 0.005s Metrics: {"bytes_written":3993924,"cfile_cache_hit":11,"cfile_cache_hit_bytes":621866,"cfile_cache_miss":185,"cfile_cache_miss_bytes":9730894,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":975,"drs_written":1,"lbm_read_time_us":10125,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":205,"lbm_write_time_us":11081,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":186,"num_input_rowsets":2,"peak_mem_usage":2231064,"rows_written":142400,"spinlock_wait_cycles":1664}
I20240503 08:13:02.299351 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.335896 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.036s user 0.021s sys 0.000s Metrics: {"bytes_written":375859,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":1335,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":13050}
I20240503 08:13:02.336069 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1665389 bytes on disk
I20240503 08:13:02.337219 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":33,"lbm_reads_lt_1ms":8}
I20240503 08:13:02.337361 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013349
I20240503 08:13:02.341885 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.182s user 0.133s sys 0.015s Metrics: {"bytes_written":4193902,"cfile_cache_hit":12,"cfile_cache_hit_bytes":790041,"cfile_cache_miss":194,"cfile_cache_miss_bytes":10092861,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":4481,"drs_written":1,"lbm_read_time_us":4758,"lbm_reads_lt_1ms":214,"lbm_write_time_us":8177,"lbm_writes_lt_1ms":195,"num_input_rowsets":2,"peak_mem_usage":2347490,"rows_written":149550,"spinlock_wait_cycles":4224}
I20240503 08:13:02.342087 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1625499 bytes on disk
I20240503 08:13:02.342242 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.403797 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.427076 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.023s user 0.014s sys 0.000s Metrics: {"bytes_written":276331,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1070,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:13:02.427561 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013187
I20240503 08:13:02.439673 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.204s user 0.146s sys 0.014s Metrics: {"bytes_written":4244403,"cfile_cache_hit":13,"cfile_cache_hit_bytes":829602,"cfile_cache_miss":193,"cfile_cache_miss_bytes":10183588,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":975,"drs_written":1,"lbm_read_time_us":5119,"lbm_reads_lt_1ms":221,"lbm_write_time_us":10494,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":196,"num_input_rowsets":2,"peak_mem_usage":2376290,"rows_written":151350}
I20240503 08:13:02.439945 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1646042 bytes on disk
I20240503 08:13:02.441601 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.455082 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.493399 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.037s user 0.014s sys 0.000s Metrics: {"bytes_written":271122,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1014,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5754,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9400}
I20240503 08:13:02.493943 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1055098 bytes of WAL
I20240503 08:13:02.494051 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:02.494091 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000005 (ops 2560-3194)
I20240503 08:13:02.494771 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:02.494956 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 86604 bytes on disk
I20240503 08:13:02.495151 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.495316 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013123
I20240503 08:13:02.610244 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.272s user 0.139s sys 0.019s Metrics: {"bytes_written":4354926,"cfile_cache_hit":12,"cfile_cache_hit_bytes":724506,"cfile_cache_miss":198,"cfile_cache_miss_bytes":10586344,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":2354,"drs_written":1,"lbm_read_time_us":10741,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":216,"lbm_write_time_us":25093,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":200,"num_input_rowsets":2,"peak_mem_usage":2439862,"rows_written":155450,"spinlock_wait_cycles":2048}
I20240503 08:13:02.610790 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.637157 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.026s user 0.011s sys 0.004s Metrics: {"bytes_written":320080,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":925,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":878,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11100}
I20240503 08:13:02.637427 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1051029 bytes of WAL
I20240503 08:13:02.637540 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:02.637603 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000005 (ops 2523-3151)
I20240503 08:13:02.638231 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:02.638376 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013169
I20240503 08:13:02.651129 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.223s user 0.154s sys 0.008s Metrics: {"bytes_written":4461705,"cfile_cache_hit":12,"cfile_cache_hit_bytes":789439,"cfile_cache_miss":202,"cfile_cache_miss_bytes":10802953,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":811,"drs_written":1,"lbm_read_time_us":6373,"lbm_reads_lt_1ms":226,"lbm_write_time_us":8743,"lbm_writes_lt_1ms":207,"num_input_rowsets":2,"peak_mem_usage":2503431,"rows_written":159150,"spinlock_wait_cycles":896}
I20240503 08:13:02.651337 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1735506 bytes on disk
I20240503 08:13:02.651468 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.681964 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.703317 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.021s user 0.013s sys 0.000s Metrics: {"bytes_written":277692,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":1029,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":755,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:02.703650 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013087
I20240503 08:13:02.733068 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.238s user 0.167s sys 0.006s Metrics: {"bytes_written":4506607,"cfile_cache_hit":11,"cfile_cache_hit_bytes":470413,"cfile_cache_miss":205,"cfile_cache_miss_bytes":11241691,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1893,"drs_written":1,"lbm_read_time_us":7387,"lbm_reads_lt_1ms":225,"lbm_write_time_us":11236,"lbm_writes_lt_1ms":209,"num_input_rowsets":2,"peak_mem_usage":2529630,"rows_written":160750,"spinlock_wait_cycles":7808}
I20240503 08:13:02.733395 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1753794 bytes on disk
I20240503 08:13:02.733587 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.733755 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.758705 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.025s user 0.013s sys 0.003s Metrics: {"bytes_written":277764,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":855,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:02.759276 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89663 bytes on disk
I20240503 08:13:02.761171 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.761448 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013070
I20240503 08:13:02.831732 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.193s user 0.155s sys 0.016s Metrics: {"bytes_written":4667593,"cfile_cache_hit":15,"cfile_cache_hit_bytes":980353,"cfile_cache_miss":209,"cfile_cache_miss_bytes":11152589,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":795,"drs_written":1,"lbm_read_time_us":5718,"lbm_reads_lt_1ms":237,"lbm_write_time_us":8598,"lbm_writes_lt_1ms":216,"num_input_rowsets":2,"peak_mem_usage":2622171,"rows_written":166550,"spinlock_wait_cycles":384}
I20240503 08:13:02.832132 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1819811 bytes on disk
I20240503 08:13:02.832510 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.884649 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.916846 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.031s user 0.014s sys 0.000s Metrics: {"bytes_written":275070,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":1209,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":823,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20240503 08:13:02.917126 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88968 bytes on disk
I20240503 08:13:02.917431 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.917655 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012916
I20240503 08:13:02.933279 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.229s user 0.148s sys 0.016s Metrics: {"bytes_written":4727371,"cfile_cache_hit":12,"cfile_cache_hit_bytes":789481,"cfile_cache_miss":214,"cfile_cache_miss_bytes":11512919,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":619,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":1035,"drs_written":1,"lbm_read_time_us":7681,"lbm_reads_lt_1ms":238,"lbm_write_time_us":15212,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":215,"num_input_rowsets":2,"peak_mem_usage":2660181,"rows_written":168800,"spinlock_wait_cycles":1536}
I20240503 08:13:02.934878 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1845082 bytes on disk
I20240503 08:13:02.935048 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.945655 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:02.965138 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.019s user 0.012s sys 0.002s Metrics: {"bytes_written":272609,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":666,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":654,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:13:02.965425 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1064087 bytes of WAL
I20240503 08:13:02.965557 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:02.965612 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000006 (ops 2908-3493)
I20240503 08:13:02.966081 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:02.966212 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 87220 bytes on disk
I20240503 08:13:02.967257 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:02.967563 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012894
I20240503 08:13:03.006392 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.245s user 0.170s sys 0.007s Metrics: {"bytes_written":4771650,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1004419,"cfile_cache_miss":212,"cfile_cache_miss_bytes":11417597,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":492,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":886,"drs_written":1,"lbm_read_time_us":7006,"lbm_reads_lt_1ms":232,"lbm_write_time_us":12636,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":219,"num_input_rowsets":2,"peak_mem_usage":2686180,"rows_written":170400,"spinlock_wait_cycles":2560,"thread_start_us":86,"threads_started":1}
I20240503 08:13:03.006806 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1863961 bytes on disk
I20240503 08:13:03.007027 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.007385 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.028760 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":286954,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":796,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":963,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20240503 08:13:03.029305 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 92413 bytes on disk
I20240503 08:13:03.029552 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.029708 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013062
I20240503 08:13:03.166682 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.249s user 0.174s sys 0.012s Metrics: {"bytes_written":4929964,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1142258,"cfile_cache_miss":221,"cfile_cache_miss_bytes":11702252,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":232,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":859,"drs_written":1,"lbm_read_time_us":10369,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":239,"lbm_write_time_us":14243,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":225,"num_input_rowsets":2,"peak_mem_usage":2778897,"rows_written":176100,"spinlock_wait_cycles":896}
I20240503 08:13:03.167707 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1930125 bytes on disk
I20240503 08:13:03.167955 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.178290 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.204828 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.026s user 0.015s sys 0.000s Metrics: {"bytes_written":276480,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":780,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2357,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9600}
I20240503 08:13:03.205370 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 88482 bytes on disk
I20240503 08:13:03.205596 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.205772 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013047
I20240503 08:13:03.259089 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.291s user 0.160s sys 0.028s Metrics: {"bytes_written":4993201,"cfile_cache_hit":13,"cfile_cache_hit_bytes":813571,"cfile_cache_miss":225,"cfile_cache_miss_bytes":12184403,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":1130,"drs_written":1,"lbm_read_time_us":9507,"lbm_reads_lt_1ms":245,"lbm_write_time_us":11878,"lbm_writes_lt_1ms":229,"num_input_rowsets":2,"peak_mem_usage":2813931,"rows_written":178250,"spinlock_wait_cycles":1408}
I20240503 08:13:03.259414 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 1955587 bytes on disk
I20240503 08:13:03.259627 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.259780 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.293125 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.033s user 0.017s sys 0.000s Metrics: {"bytes_written":310926,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":860,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3509,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10750}
I20240503 08:13:03.293498 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 99336 bytes on disk
I20240503 08:13:03.293673 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.293852 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.013011
I20240503 08:13:03.307698 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.278s user 0.178s sys 0.019s Metrics: {"bytes_written":5052380,"cfile_cache_hit":13,"cfile_cache_hit_bytes":965654,"cfile_cache_miss":227,"cfile_cache_miss_bytes":12189462,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":9134,"lbm_reads_lt_1ms":247,"lbm_write_time_us":13761,"lbm_writes_lt_1ms":232,"num_input_rowsets":2,"peak_mem_usage":2848320,"rows_written":180400}
I20240503 08:13:03.308908 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.327548 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":301524,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":1693,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":826,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10400}
I20240503 08:13:03.327872 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012900
I20240503 08:13:03.450661 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.245s user 0.166s sys 0.030s Metrics: {"bytes_written":5197354,"cfile_cache_hit":12,"cfile_cache_hit_bytes":645519,"cfile_cache_miss":234,"cfile_cache_miss_bytes":12905021,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":913,"drs_written":1,"lbm_read_time_us":8528,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":253,"lbm_write_time_us":11524,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":236,"num_input_rowsets":2,"peak_mem_usage":2934837,"rows_written":185700,"spinlock_wait_cycles":2432}
I20240503 08:13:03.451159 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2041334 bytes on disk
I20240503 08:13:03.451440 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.472692 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.498068 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.025s user 0.011s sys 0.003s Metrics: {"bytes_written":275126,"cfile_init":1,"dirs.queue_time_us":124,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":778,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5071,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9550,"spinlock_wait_cycles":384}
I20240503 08:13:03.498570 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1057868 bytes of WAL
I20240503 08:13:03.499812 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:03.499866 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000006 (ops 3152-3787)
I20240503 08:13:03.500303 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:03.500460 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012849
I20240503 08:13:03.599659 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.305s user 0.195s sys 0.005s Metrics: {"bytes_written":5289842,"cfile_cache_hit":11,"cfile_cache_hit_bytes":500671,"cfile_cache_miss":237,"cfile_cache_miss_bytes":13293149,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":951,"drs_written":1,"lbm_read_time_us":9329,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":256,"lbm_write_time_us":12438,"lbm_writes_lt_1ms":241,"num_input_rowsets":2,"peak_mem_usage":2987311,"rows_written":189000,"spinlock_wait_cycles":2048}
I20240503 08:13:03.599992 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2078894 bytes on disk
I20240503 08:13:03.600159 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.600291 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.625875 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.298s user 0.170s sys 0.026s Metrics: {"bytes_written":5339588,"cfile_cache_hit":13,"cfile_cache_hit_bytes":749005,"cfile_cache_miss":239,"cfile_cache_miss_bytes":13178109,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1029,"drs_written":1,"lbm_read_time_us":8340,"lbm_reads_lt_1ms":267,"lbm_write_time_us":19539,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":239,"num_input_rowsets":2,"peak_mem_usage":3016910,"rows_written":190800,"spinlock_wait_cycles":3456}
I20240503 08:13:03.626317 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2099617 bytes on disk
I20240503 08:13:03.626511 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.637254 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.651013 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.051s user 0.017s sys 0.000s Metrics: {"bytes_written":302900,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":7431,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":837,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10450}
I20240503 08:13:03.651729 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012855
I20240503 08:13:03.672528 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.035s user 0.014s sys 0.004s Metrics: {"bytes_written":272522,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1078,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:13:03.672767 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1058939 bytes of WAL
I20240503 08:13:03.672914 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:03.672993 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000006 (ops 3195-3836)
I20240503 08:13:03.673730 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:03.674831 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 86920 bytes on disk
I20240503 08:13:03.675024 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.675335 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012846
I20240503 08:13:03.774233 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.273s user 0.199s sys 0.008s Metrics: {"bytes_written":5465339,"cfile_cache_hit":12,"cfile_cache_hit_bytes":504207,"cfile_cache_miss":244,"cfile_cache_miss_bytes":13750013,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1161,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":798,"drs_written":1,"lbm_read_time_us":7142,"lbm_reads_lt_1ms":268,"lbm_write_time_us":11808,"lbm_writes_lt_1ms":249,"num_input_rowsets":2,"peak_mem_usage":3090063,"rows_written":195250,"spinlock_wait_cycles":3588224}
I20240503 08:13:03.774515 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2150682 bytes on disk
I20240503 08:13:03.774819 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20240503 08:13:03.795629 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.813539 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":280287,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":797,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20240503 08:13:03.813865 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012799
I20240503 08:13:03.944514 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.292s user 0.189s sys 0.021s Metrics: {"bytes_written":5578099,"cfile_cache_hit":11,"cfile_cache_hit_bytes":494212,"cfile_cache_miss":249,"cfile_cache_miss_bytes":14067748,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1000,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1430,"drs_written":1,"lbm_read_time_us":9821,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":272,"lbm_write_time_us":14327,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":254,"num_input_rowsets":2,"peak_mem_usage":3157411,"rows_written":199450,"spinlock_wait_cycles":768}
I20240503 08:13:03.945755 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:03.964721 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.019s user 0.012s sys 0.003s Metrics: {"bytes_written":313633,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":771,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10850}
I20240503 08:13:03.964974 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2300558 bytes on disk
I20240503 08:13:03.965152 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":38,"lbm_reads_lt_1ms":8}
I20240503 08:13:03.965298 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012744
I20240503 08:13:04.044446 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.369s user 0.204s sys 0.013s Metrics: {"bytes_written":5602465,"cfile_cache_hit":14,"cfile_cache_hit_bytes":839620,"cfile_cache_miss":248,"cfile_cache_miss_bytes":13783968,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":589,"drs_written":1,"lbm_read_time_us":8937,"lbm_reads_lt_1ms":268,"lbm_write_time_us":18874,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":256,"num_input_rowsets":2,"peak_mem_usage":3171410,"rows_written":200250,"spinlock_wait_cycles":3456}
I20240503 08:13:04.044792 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2208661 bytes on disk
I20240503 08:13:04.044996 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:13:04.045141 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:04.083309 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.038s user 0.018s sys 0.003s Metrics: {"bytes_written":373049,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":978,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2665,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":12950}
I20240503 08:13:04.085094 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012714
I20240503 08:13:04.202847 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.389s user 0.196s sys 0.023s Metrics: {"bytes_written":5733267,"cfile_cache_hit":11,"cfile_cache_hit_bytes":478723,"cfile_cache_miss":257,"cfile_cache_miss_bytes":14497167,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":2354,"drs_written":1,"lbm_read_time_us":15628,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":277,"lbm_write_time_us":15465,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":259,"num_input_rowsets":2,"peak_mem_usage":3248293,"rows_written":205000,"spinlock_wait_cycles":287488}
I20240503 08:13:04.203270 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:04.221709 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.256s user 0.195s sys 0.024s Metrics: {"bytes_written":5878962,"cfile_cache_hit":10,"cfile_cache_hit_bytes":469532,"cfile_cache_miss":266,"cfile_cache_miss_bytes":14891894,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":822,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1188,"drs_written":1,"lbm_read_time_us":10779,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":285,"lbm_write_time_us":11728,"lbm_writes_lt_1ms":268,"num_input_rowsets":2,"peak_mem_usage":3333311,"rows_written":210300,"spinlock_wait_cycles":113408}
I20240503 08:13:04.222051 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1057183 bytes of WAL
I20240503 08:13:04.222180 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:04.222234 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000007 (ops 3494-4075)
I20240503 08:13:04.222746 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:04.222898 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2324641 bytes on disk
I20240503 08:13:04.223073 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:13:04.237426 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.034s user 0.022s sys 0.000s Metrics: {"bytes_written":399584,"cfile_init":1,"dirs.queue_time_us":31,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1044,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":13800}
I20240503 08:13:04.240413 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2391759 bytes on disk
I20240503 08:13:04.244742 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20240503 08:13:04.245244 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:04.245393 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012602
I20240503 08:13:04.272614 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.027s user 0.013s sys 0.003s Metrics: {"bytes_written":272569,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1073,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20240503 08:13:04.272778 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 86912 bytes on disk
I20240503 08:13:04.273415 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20240503 08:13:04.273574 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012590
I20240503 08:13:04.506134 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.421s user 0.223s sys 0.014s Metrics: {"bytes_written":5960450,"cfile_cache_hit":10,"cfile_cache_hit_bytes":510002,"cfile_cache_miss":270,"cfile_cache_miss_bytes":15061578,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":657,"dirs.run_wall_time_us":6899,"drs_written":1,"lbm_read_time_us":12343,"lbm_reads_lt_1ms":294,"lbm_write_time_us":20737,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":269,"num_input_rowsets":2,"peak_mem_usage":3378333,"rows_written":213200,"spinlock_wait_cycles":4369664}
I20240503 08:13:04.507444 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:04.558599 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.051s user 0.025s sys 0.000s Metrics: {"bytes_written":497165,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":937,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1281,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":17350}
I20240503 08:13:04.559585 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1060769 bytes of WAL
I20240503 08:13:04.559767 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:04.559857 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000007 (ops 3837-4464)
I20240503 08:13:04.560900 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:04.561059 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012541
I20240503 08:13:04.807559 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.562s user 0.208s sys 0.029s Metrics: {"bytes_written":6116608,"cfile_cache_hit":11,"cfile_cache_hit_bytes":568576,"cfile_cache_miss":275,"cfile_cache_miss_bytes":15416296,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":944,"drs_written":1,"lbm_read_time_us":26844,"lbm_reads_1-10_ms":5,"lbm_reads_lt_1ms":290,"lbm_write_time_us":30044,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":272,"num_input_rowsets":2,"peak_mem_usage":3468474,"rows_written":218800,"spinlock_wait_cycles":7168,"thread_start_us":246,"threads_started":5}
I20240503 08:13:04.808894 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.937278
I20240503 08:13:04.850844 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.042s user 0.030s sys 0.003s Metrics: {"bytes_written":760519,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":859,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1833,"lbm_writes_lt_1ms":51,"peak_mem_usage":0,"rows_written":26700}
I20240503 08:13:04.851150 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1049716 bytes of WAL
I20240503 08:13:04.851320 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:04.851418 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000007 (ops 3788-4406)
I20240503 08:13:04.851851 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:04.852046 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2682969 bytes on disk
I20240503 08:13:04.852249 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20240503 08:13:04.852416 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012410
I20240503 08:13:04.943603 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.670s user 0.212s sys 0.029s Metrics: {"bytes_written":6145181,"cfile_cache_hit":12,"cfile_cache_hit_bytes":507545,"cfile_cache_miss":278,"cfile_cache_miss_bytes":15554269,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":15744,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":734,"drs_written":1,"lbm_read_time_us":18971,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":297,"lbm_write_time_us":37530,"lbm_writes_1-10_ms":10,"lbm_writes_lt_1ms":268,"num_input_rowsets":2,"peak_mem_usage":3488260,"rows_written":219750,"thread_start_us":106,"threads_started":1}
I20240503 08:13:04.944973 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=2.929465
I20240503 08:13:04.993592 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.432s user 0.212s sys 0.037s Metrics: {"bytes_written":6442396,"cfile_cache_hit":14,"cfile_cache_hit_bytes":958209,"cfile_cache_miss":286,"cfile_cache_miss_bytes":15879757,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":316,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":801,"drs_written":1,"lbm_read_time_us":15535,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":312,"lbm_write_time_us":27953,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":286,"num_input_rowsets":2,"peak_mem_usage":3655848,"rows_written":230550,"spinlock_wait_cycles":87296,"thread_start_us":143,"threads_started":2}
I20240503 08:13:04.993929 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.937278
I20240503 08:13:05.027307 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.033s user 0.023s sys 0.003s Metrics: {"bytes_written":624803,"cfile_init":1,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":888,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1253,"lbm_writes_lt_1ms":46,"peak_mem_usage":0,"rows_written":21950}
I20240503 08:13:05.027582 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2772573 bytes on disk
I20240503 08:13:05.027779 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":35,"lbm_reads_lt_1ms":8}
I20240503 08:13:05.028187 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012342
I20240503 08:13:05.035182 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.090s user 0.039s sys 0.000s Metrics: {"bytes_written":884139,"cfile_init":1,"dirs.queue_time_us":5543,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":715,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10528,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":56,"peak_mem_usage":0,"rows_written":31100}
I20240503 08:13:05.035431 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1084315 bytes of WAL
I20240503 08:13:05.035529 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:05.035580 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000008 (ops 4076-4672)
I20240503 08:13:05.036799 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:05.037083 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2740817 bytes on disk
I20240503 08:13:05.037303 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":42,"lbm_reads_lt_1ms":8}
I20240503 08:13:05.037801 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012357
I20240503 08:13:05.178007 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.325s user 0.217s sys 0.041s Metrics: {"bytes_written":6859113,"cfile_cache_hit":12,"cfile_cache_hit_bytes":640844,"cfile_cache_miss":304,"cfile_cache_miss_bytes":17291410,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":872,"drs_written":1,"lbm_read_time_us":8347,"lbm_reads_lt_1ms":324,"lbm_write_time_us":13977,"lbm_writes_lt_1ms":308,"num_input_rowsets":2,"peak_mem_usage":3889405,"rows_written":245500,"spinlock_wait_cycles":5120}
I20240503 08:13:05.178265 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2730449 bytes on disk
I20240503 08:13:05.178488 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.178645 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:05.218647 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.037s user 0.018s sys 0.000s Metrics: {"bytes_written":334638,"cfile_init":1,"dirs.queue_time_us":242,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":983,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2048,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":11650}
I20240503 08:13:05.219365 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1059287 bytes of WAL
I20240503 08:13:05.219446 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:05.219475 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000008 (ops 4407-5017)
I20240503 08:13:05.219805 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:05.219961 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012280
I20240503 08:13:05.315086 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.287s user 0.248s sys 0.020s Metrics: {"bytes_written":7051145,"cfile_cache_hit":10,"cfile_cache_hit_bytes":527447,"cfile_cache_miss":316,"cfile_cache_miss_bytes":17931203,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":288,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":8454,"lbm_reads_lt_1ms":336,"lbm_write_time_us":13996,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":316,"num_input_rowsets":2,"peak_mem_usage":4007898,"rows_written":252500,"spinlock_wait_cycles":2784384}
I20240503 08:13:05.315459 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2812694 bytes on disk
I20240503 08:13:05.315665 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.366828 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:05.388720 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.022s user 0.014s sys 0.000s Metrics: {"bytes_written":277756,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":806,"drs_written":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4,"lbm_write_time_us":650,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20240503 08:13:05.389112 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1055065 bytes of WAL
I20240503 08:13:05.389256 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:05.389335 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000008 (ops 4465-5078)
I20240503 08:13:05.389834 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:05.389956 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 89050 bytes on disk
I20240503 08:13:05.390103 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.390233 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011994
I20240503 08:13:05.406996 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.369s user 0.244s sys 0.029s Metrics: {"bytes_written":7006197,"cfile_cache_hit":12,"cfile_cache_hit_bytes":677024,"cfile_cache_miss":310,"cfile_cache_miss_bytes":17642368,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1142,"drs_written":1,"lbm_read_time_us":9732,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":329,"lbm_write_time_us":19189,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":311,"num_input_rowsets":2,"peak_mem_usage":3971936,"rows_written":250850,"spinlock_wait_cycles":4864}
I20240503 08:13:05.407858 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2793090 bytes on disk
I20240503 08:13:05.414836 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.415088 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:05.438400 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.023s user 0.015s sys 0.002s Metrics: {"bytes_written":355831,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":883,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":881,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12300}
I20240503 08:13:05.438747 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1057149 bytes of WAL
I20240503 08:13:05.438884 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:05.438966 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000009 (ops 4673-5254)
I20240503 08:13:05.439513 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:05.439657 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 114653 bytes on disk
I20240503 08:13:05.440147 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":330,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.440335 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012221
I20240503 08:13:05.690662 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.471s user 0.250s sys 0.020s Metrics: {"bytes_written":7181569,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1102791,"cfile_cache_miss":315,"cfile_cache_miss_bytes":17716299,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1543,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":810,"drs_written":1,"lbm_read_time_us":15528,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":337,"lbm_write_time_us":26112,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":317,"mutex_wait_us":410,"num_input_rowsets":2,"peak_mem_usage":4093771,"rows_written":257150,"spinlock_wait_cycles":384,"thread_start_us":165,"threads_started":4}
I20240503 08:13:05.690912 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2865825 bytes on disk
I20240503 08:13:05.692236 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.692405 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:05.731451 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.291s user 0.244s sys 0.024s Metrics: {"bytes_written":7348939,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1171128,"cfile_cache_miss":323,"cfile_cache_miss_bytes":18089892,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":828,"drs_written":1,"lbm_read_time_us":11525,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":342,"lbm_write_time_us":14743,"lbm_writes_lt_1ms":331,"num_input_rowsets":2,"peak_mem_usage":4190613,"rows_written":263150,"spinlock_wait_cycles":2944}
I20240503 08:13:05.731779 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2936215 bytes on disk
I20240503 08:13:05.731962 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.732106 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:05.745211 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.051s user 0.014s sys 0.007s Metrics: {"bytes_written":437229,"cfile_init":1,"dirs.queue_time_us":53,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2080,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":15300}
I20240503 08:13:05.745431 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.012049
I20240503 08:13:05.755175 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.023s user 0.017s sys 0.000s Metrics: {"bytes_written":317554,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":2593,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11000}
I20240503 08:13:05.755448 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 101726 bytes on disk
I20240503 08:13:05.755636 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.755770 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011825
I20240503 08:13:05.812412 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.422s user 0.264s sys 0.005s Metrics: {"bytes_written":7322835,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1092427,"cfile_cache_miss":323,"cfile_cache_miss_bytes":18098565,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1490,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":17570,"drs_written":1,"lbm_read_time_us":12921,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":341,"lbm_write_time_us":22194,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":327,"num_input_rowsets":2,"peak_mem_usage":4177014,"rows_written":262150,"spinlock_wait_cycles":3407488}
I20240503 08:13:05.812749 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 2924361 bytes on disk
I20240503 08:13:05.813098 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.813242 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:05.860466 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.047s user 0.016s sys 0.004s Metrics: {"bytes_written":409590,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":977,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2471,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14200}
I20240503 08:13:05.860778 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 133158 bytes on disk
I20240503 08:13:05.860965 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20240503 08:13:05.861107 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011940
I20240503 08:13:06.159303 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.403s user 0.274s sys 0.009s Metrics: {"bytes_written":7654607,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1136747,"cfile_cache_miss":342,"cfile_cache_miss_bytes":18944125,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":203,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":935,"drs_written":1,"lbm_read_time_us":17650,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":358,"lbm_write_time_us":20513,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":338,"num_input_rowsets":2,"peak_mem_usage":4370647,"rows_written":274150,"spinlock_wait_cycles":5760}
I20240503 08:13:06.159657 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3063626 bytes on disk
I20240503 08:13:06.159839 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20240503 08:13:06.160218 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:06.194238 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.033s user 0.019s sys 0.003s Metrics: {"bytes_written":513480,"cfile_init":1,"dirs.queue_time_us":391,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":1338,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1745,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18000}
I20240503 08:13:06.194540 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1071610 bytes of WAL
I20240503 08:13:06.194729 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:06.194813 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000010 (ops 5255-5844)
I20240503 08:13:06.195348 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:06.195501 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 171771 bytes on disk
I20240503 08:13:06.195732 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4}
I20240503 08:13:06.195912 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011870
I20240503 08:13:06.318142 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.566s user 0.251s sys 0.040s Metrics: {"bytes_written":7607753,"cfile_cache_hit":13,"cfile_cache_hit_bytes":905975,"cfile_cache_miss":337,"cfile_cache_miss_bytes":19031069,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":826,"drs_written":1,"lbm_read_time_us":22241,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":357,"lbm_write_time_us":26510,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":338,"num_input_rowsets":2,"peak_mem_usage":4338721,"rows_written":272450,"spinlock_wait_cycles":8705408}
I20240503 08:13:06.318926 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.937278
I20240503 08:13:06.377126 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.058s user 0.032s sys 0.003s Metrics: {"bytes_written":725444,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":1146,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3274,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":48,"peak_mem_usage":0,"rows_written":25500}
I20240503 08:13:06.377976 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1059882 bytes of WAL
I20240503 08:13:06.378413 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:06.378508 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000009 (ops 5018-5639)
I20240503 08:13:06.379060 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:06.379220 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3291175 bytes on disk
I20240503 08:13:06.379976 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":558,"lbm_reads_lt_1ms":8}
I20240503 08:13:06.380147 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011865
I20240503 08:13:06.434526 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.573s user 0.241s sys 0.056s Metrics: {"bytes_written":7716208,"cfile_cache_hit":12,"cfile_cache_hit_bytes":736904,"cfile_cache_miss":344,"cfile_cache_miss_bytes":19498512,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":830,"drs_written":1,"lbm_read_time_us":15446,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":362,"lbm_write_time_us":34453,"lbm_writes_1-10_ms":8,"lbm_writes_lt_1ms":337,"num_input_rowsets":2,"peak_mem_usage":4403196,"rows_written":276350,"spinlock_wait_cycles":13509248,"thread_start_us":221,"threads_started":5}
I20240503 08:13:06.434893 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3088627 bytes on disk
I20240503 08:13:06.436703 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20240503 08:13:06.436906 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.937278
I20240503 08:13:06.492697 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.056s user 0.032s sys 0.000s Metrics: {"bytes_written":705642,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":848,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3533,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":48,"peak_mem_usage":0,"rows_written":24750}
I20240503 08:13:06.493006 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1056121 bytes of WAL
I20240503 08:13:06.493180 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:06.493238 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000009 (ops 5079-5702)
I20240503 08:13:06.493697 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:06.493829 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 238237 bytes on disk
I20240503 08:13:06.494046 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20240503 08:13:06.494199 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011832
I20240503 08:13:06.542050 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.346s user 0.286s sys 0.018s Metrics: {"bytes_written":8152126,"cfile_cache_hit":12,"cfile_cache_hit_bytes":857802,"cfile_cache_miss":362,"cfile_cache_miss_bytes":20530530,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":366,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":849,"drs_written":1,"lbm_read_time_us":9598,"lbm_reads_lt_1ms":382,"lbm_write_time_us":21082,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":363,"num_input_rowsets":2,"peak_mem_usage":4656126,"rows_written":292150,"spinlock_wait_cycles":3328}
I20240503 08:13:06.542378 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:06.562973 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.020s user 0.013s sys 0.004s Metrics: {"bytes_written":342374,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":740,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":795,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11950}
I20240503 08:13:06.563269 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011796
I20240503 08:13:06.752277 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.372s user 0.267s sys 0.035s Metrics: {"bytes_written":8314942,"cfile_cache_hit":11,"cfile_cache_hit_bytes":693464,"cfile_cache_miss":367,"cfile_cache_miss_bytes":21111932,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":639,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":815,"drs_written":1,"lbm_read_time_us":11369,"lbm_reads_lt_1ms":387,"lbm_write_time_us":20097,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":369,"mutex_wait_us":1,"num_input_rowsets":2,"peak_mem_usage":4742861,"rows_written":297950,"spinlock_wait_cycles":2688}
I20240503 08:13:06.752766 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3340133 bytes on disk
I20240503 08:13:06.752979 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:06.773530 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:06.799659 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.026s user 0.013s sys 0.002s Metrics: {"bytes_written":279111,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":938,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1894,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20240503 08:13:06.800189 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011711
I20240503 08:13:06.871722 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.308s user 0.272s sys 0.032s Metrics: {"bytes_written":8484318,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1022321,"cfile_cache_miss":374,"cfile_cache_miss_bytes":21266539,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":698,"drs_written":1,"lbm_read_time_us":9864,"lbm_reads_lt_1ms":402,"lbm_write_time_us":14820,"lbm_writes_lt_1ms":377,"num_input_rowsets":2,"peak_mem_usage":4856358,"rows_written":304100}
I20240503 08:13:06.872041 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:06.887037 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.015s user 0.011s sys 0.003s Metrics: {"bytes_written":281524,"cfile_init":1,"dirs.queue_time_us":30,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20240503 08:13:06.887732 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3501058 bytes on disk
I20240503 08:13:06.887939 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":41,"lbm_reads_lt_1ms":8}
I20240503 08:13:06.888649 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011667
I20240503 08:13:06.922866 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.428s user 0.272s sys 0.041s Metrics: {"bytes_written":8403944,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1427476,"cfile_cache_miss":366,"cfile_cache_miss_bytes":20610142,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":863,"drs_written":1,"lbm_read_time_us":12877,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":385,"lbm_write_time_us":19692,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":374,"num_input_rowsets":2,"peak_mem_usage":4794799,"rows_written":301100,"spinlock_wait_cycles":1152}
I20240503 08:13:06.923214 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:06.962811 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.039s user 0.021s sys 0.001s Metrics: {"bytes_written":424717,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":822,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2376,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":14800}
I20240503 08:13:06.964787 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3515676 bytes on disk
I20240503 08:13:06.964975 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":35,"lbm_reads_lt_1ms":8}
I20240503 08:13:06.965504 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011647
I20240503 08:13:07.203368 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.315s user 0.287s sys 0.024s Metrics: {"bytes_written":8755524,"cfile_cache_hit":12,"cfile_cache_hit_bytes":645624,"cfile_cache_miss":386,"cfile_cache_miss_bytes":22370688,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1093,"drs_written":1,"lbm_read_time_us":10242,"lbm_reads_lt_1ms":406,"lbm_write_time_us":15581,"lbm_writes_lt_1ms":389,"num_input_rowsets":2,"peak_mem_usage":5017749,"rows_written":313900,"spinlock_wait_cycles":4224}
I20240503 08:13:07.204761 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3523871 bytes on disk
I20240503 08:13:07.205209 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:13:07.205385 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:07.225426 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.020s user 0.017s sys 0.000s Metrics: {"bytes_written":350109,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":2142,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":748,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12100}
I20240503 08:13:07.226290 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1056829 bytes of WAL
I20240503 08:13:07.226748 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:07.226838 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000011 (ops 5845-6426)
I20240503 08:13:07.227458 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:07.227635 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011218
I20240503 08:13:07.362329 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.562s user 0.279s sys 0.033s Metrics: {"bytes_written":8580542,"cfile_cache_hit":13,"cfile_cache_hit_bytes":808774,"cfile_cache_miss":379,"cfile_cache_miss_bytes":21745912,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":365,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":868,"drs_written":1,"lbm_read_time_us":24078,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":400,"lbm_write_time_us":27957,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":380,"num_input_rowsets":2,"peak_mem_usage":4916292,"rows_written":307650,"spinlock_wait_cycles":1536,"thread_start_us":242,"threads_started":5}
I20240503 08:13:07.362838 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.937278
I20240503 08:13:07.409158 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.046s user 0.024s sys 0.004s Metrics: {"bytes_written":654802,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":897,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1513,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":22950}
I20240503 08:13:07.409449 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1120690 bytes of WAL
I20240503 08:13:07.409641 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:07.409720 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000010 (ops 5640-6323)
I20240503 08:13:07.410210 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:07.410465 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011501
I20240503 08:13:07.458344 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.493s user 0.303s sys 0.025s Metrics: {"bytes_written":8810660,"cfile_cache_hit":14,"cfile_cache_hit_bytes":933007,"cfile_cache_miss":388,"cfile_cache_miss_bytes":22221719,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":14193,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":407,"lbm_write_time_us":19722,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":390,"num_input_rowsets":2,"peak_mem_usage":5045241,"rows_written":315900,"spinlock_wait_cycles":384,"thread_start_us":681,"threads_started":5}
I20240503 08:13:07.458788 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:07.502173 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.043s user 0.024s sys 0.000s Metrics: {"bytes_written":486547,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":963,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1271,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":17100}
I20240503 08:13:07.503265 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1060357 bytes of WAL
I20240503 08:13:07.503432 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:07.503513 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000010 (ops 5703-6351)
I20240503 08:13:07.503942 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:07.504065 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3708950 bytes on disk
I20240503 08:13:07.504678 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":429,"lbm_reads_lt_1ms":8}
I20240503 08:13:07.504935 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011152
I20240503 08:13:07.618496 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.391s user 0.304s sys 0.027s Metrics: {"bytes_written":9090214,"cfile_cache_hit":12,"cfile_cache_hit_bytes":822414,"cfile_cache_miss":400,"cfile_cache_miss_bytes":23078760,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":919,"drs_written":1,"lbm_read_time_us":11609,"lbm_reads_lt_1ms":424,"lbm_write_time_us":17586,"lbm_writes_lt_1ms":403,"num_input_rowsets":2,"peak_mem_usage":5212052,"rows_written":326000,"spinlock_wait_cycles":6144}
I20240503 08:13:07.619681 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:07.644205 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.024s user 0.017s sys 0.000s Metrics: {"bytes_written":334532,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":2927,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":835,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11650}
I20240503 08:13:07.644551 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3773203 bytes on disk
I20240503 08:13:07.644779 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":43,"lbm_reads_lt_1ms":8}
I20240503 08:13:07.644999 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011132
I20240503 08:13:07.877171 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.467s user 0.301s sys 0.031s Metrics: {"bytes_written":9220239,"cfile_cache_hit":13,"cfile_cache_hit_bytes":924673,"cfile_cache_miss":405,"cfile_cache_miss_bytes":23296221,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":87,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":807,"drs_written":1,"lbm_read_time_us":17217,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":431,"lbm_write_time_us":27273,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":408,"num_input_rowsets":2,"peak_mem_usage":5275940,"rows_written":330600,"spinlock_wait_cycles":1280,"thread_start_us":121,"threads_started":3}
I20240503 08:13:07.877528 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:07.906870 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.029s user 0.020s sys 0.002s Metrics: {"bytes_written":451452,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":770,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1182,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15700}
I20240503 08:13:07.907660 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 3866711 bytes on disk
I20240503 08:13:07.908336 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":38,"lbm_reads_lt_1ms":8}
I20240503 08:13:07.908531 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011044
I20240503 08:13:08.070552 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.425s user 0.303s sys 0.038s Metrics: {"bytes_written":9412560,"cfile_cache_hit":13,"cfile_cache_hit_bytes":849307,"cfile_cache_miss":413,"cfile_cache_miss_bytes":23913775,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":920,"drs_written":1,"lbm_read_time_us":13823,"lbm_reads_lt_1ms":433,"lbm_write_time_us":21085,"lbm_writes_lt_1ms":419,"num_input_rowsets":2,"peak_mem_usage":5401806,"rows_written":337650,"spinlock_wait_cycles":4992}
I20240503 08:13:08.071015 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:08.096987 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.026s user 0.021s sys 0.000s Metrics: {"bytes_written":444004,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1089,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15400}
I20240503 08:13:08.097342 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1057255 bytes of WAL
I20240503 08:13:08.097610 28457 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:08.097751 28457 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 346fb438dba74534836cebb489aa44d2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-0-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000012 (ops 6427-7008)
I20240503 08:13:08.098253 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:08.098659 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.010870
I20240503 08:13:08.153265 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.648s user 0.309s sys 0.039s Metrics: {"bytes_written":9284116,"cfile_cache_hit":12,"cfile_cache_hit_bytes":743514,"cfile_cache_miss":408,"cfile_cache_miss_bytes":23660702,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":19976,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":2032,"drs_written":1,"lbm_read_time_us":22254,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":426,"lbm_write_time_us":31717,"lbm_writes_1-10_ms":6,"lbm_writes_lt_1ms":407,"num_input_rowsets":2,"peak_mem_usage":5320792,"rows_written":333000,"spinlock_wait_cycles":4096,"thread_start_us":2674,"threads_started":6}
I20240503 08:13:08.153533 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.937278
I20240503 08:13:08.207310 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.053s user 0.024s sys 0.005s Metrics: {"bytes_written":636667,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":883,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1297,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":22250}
I20240503 08:13:08.207741 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1057512 bytes of WAL
I20240503 08:13:08.208492 28533 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:08.208590 28533 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P a215981f3cf94161ba25daa9e8b3f7aa: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-1-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000011 (ops 6352-6985)
I20240503 08:13:08.210590 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:08.210785 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011095
I20240503 08:13:08.357218 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.449s user 0.315s sys 0.036s Metrics: {"bytes_written":9654117,"cfile_cache_hit":13,"cfile_cache_hit_bytes":786947,"cfile_cache_miss":427,"cfile_cache_miss_bytes":24607591,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":224,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":938,"drs_written":1,"lbm_read_time_us":15588,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":446,"lbm_write_time_us":23764,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":427,"num_input_rowsets":2,"peak_mem_usage":5538300,"rows_written":346300,"spinlock_wait_cycles":21888}
I20240503 08:13:08.357604 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:08.412132 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.054s user 0.015s sys 0.008s Metrics: {"bytes_written":424799,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":880,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3089,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14800}
I20240503 08:13:08.412452 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3): free 1054042 bytes of WAL
I20240503 08:13:08.414909 28608 log_reader.cc:386] T ed79d2ecddc7447eadcdf1b581d723b3: removed 1 log segments from log reader
I20240503 08:13:08.416036 28608 log.cc:1076] T ed79d2ecddc7447eadcdf1b581d723b3 P 77d5740a6aa74347be19f5b66d7442f2: Deleting log segment in path: /tmp/dist-test-taskwTO9XN/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_1.1714723659965653-17235-0/minicluster-data/ts-2-root/wals/ed79d2ecddc7447eadcdf1b581d723b3/wal-000000011 (ops 6324-6945)
I20240503 08:13:08.417531 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: LogGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {}
I20240503 08:13:08.417855 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 4039093 bytes on disk
I20240503 08:13:08.419682 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20240503 08:13:08.419858 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011020
I20240503 08:13:08.505312 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.406s user 0.306s sys 0.044s Metrics: {"bytes_written":9839277,"cfile_cache_hit":10,"cfile_cache_hit_bytes":528723,"cfile_cache_miss":440,"cfile_cache_miss_bytes":25363917,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":533,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":829,"drs_written":1,"lbm_read_time_us":15254,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":467,"lbm_write_time_us":20682,"lbm_writes_lt_1ms":436,"num_input_rowsets":2,"peak_mem_usage":5648281,"rows_written":353050,"spinlock_wait_cycles":899072}
I20240503 08:13:08.505789 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:08.531620 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.026s user 0.018s sys 0.000s Metrics: {"bytes_written":362472,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":710,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1026,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12550}
I20240503 08:13:08.531963 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 4095603 bytes on disk
I20240503 08:13:08.532166 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20240503 08:13:08.532366 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.011032
I20240503 08:13:08.752343 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.541s user 0.335s sys 0.036s Metrics: {"bytes_written":9899297,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1246775,"cfile_cache_miss":435,"cfile_cache_miss_bytes":24796287,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1083,"drs_written":1,"lbm_read_time_us":20707,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":461,"lbm_write_time_us":29214,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":433,"num_input_rowsets":2,"peak_mem_usage":5677173,"rows_written":355250,"spinlock_wait_cycles":52480}
I20240503 08:13:08.752678 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 4004122 bytes on disk
I20240503 08:13:08.753083 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20240503 08:13:08.753397 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:08.784921 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.031s user 0.017s sys 0.008s Metrics: {"bytes_written":503387,"cfile_init":1,"dirs.queue_time_us":662,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":2328,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1149,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17600}
I20240503 08:13:08.785199 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 166898 bytes on disk
I20240503 08:13:08.785930 28533 maintenance_manager.cc:643] P a215981f3cf94161ba25daa9e8b3f7aa: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20240503 08:13:08.786111 28593 maintenance_manager.cc:419] P a215981f3cf94161ba25daa9e8b3f7aa: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.010912
I20240503 08:13:08.953058 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.421s user 0.323s sys 0.045s Metrics: {"bytes_written":10184676,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1353542,"cfile_cache_miss":444,"cfile_cache_miss_bytes":25468274,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1176,"drs_written":1,"lbm_read_time_us":14312,"lbm_reads_lt_1ms":464,"lbm_write_time_us":21575,"lbm_writes_lt_1ms":451,"num_input_rowsets":2,"peak_mem_usage":5855063,"rows_written":365600,"spinlock_wait_cycles":3712}
I20240503 08:13:08.954228 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:08.983312 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.029s user 0.009s sys 0.007s Metrics: {"bytes_written":362509,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":701,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8377,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":12550}
I20240503 08:13:08.983520 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3): 4241352 bytes on disk
I20240503 08:13:08.983801 28457 maintenance_manager.cc:643] P 346fb438dba74534836cebb489aa44d2: UndoDeltaBlockGCOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":45,"lbm_reads_lt_1ms":8}
I20240503 08:13:08.983922 28517 maintenance_manager.cc:419] P 346fb438dba74534836cebb489aa44d2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.010974
I20240503 08:13:08.987700 17235 test_util.cc:268] Using random seed: 1245732657
I20240503 08:13:08.991638 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.571s user 0.320s sys 0.052s Metrics: {"bytes_written":10064072,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1223954,"cfile_cache_miss":443,"cfile_cache_miss_bytes":25263892,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":923,"drs_written":1,"lbm_read_time_us":17192,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":461,"lbm_write_time_us":29819,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":442,"num_input_rowsets":2,"peak_mem_usage":5779689,"rows_written":361100,"spinlock_wait_cycles":384,"thread_start_us":243,"threads_started":5}
I20240503 08:13:08.991868 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.000000
I20240503 08:13:09.003861 17235 test_workload.cc:401] TestWorkload: Skipping table creation because table test-workload already exists
I20240503 08:13:09.014919 28608 maintenance_manager.cc:643] P 77d5740a6aa74347be19f5b66d7442f2: FlushMRSOp(ed79d2ecddc7447eadcdf1b581d723b3) complete. Timing: real 0.023s user 0.018s sys 0.005s Metrics: {"bytes_written":493291,"cfile_init":1,"dirs.queue_time_us":29,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1115,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":17200}
I20240503 08:13:09.015429 28668 maintenance_manager.cc:419] P 77d5740a6aa74347be19f5b66d7442f2: Scheduling CompactRowSetsOp(ed79d2ecddc7447eadcdf1b581d723b3): perf score=1.010974
F20240503 08:13:09.042958 28866 test_workload.cc:301] Already present: key already present
*** Check failure stack trace: ***
*** Aborted at 1714723989 (unix time) try "date -d @1714723989" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800004353) received by PID 17235 (TID 0x7f741bfde700) from PID 17235; stack trace: ***
F20240503 08:13:09.051311 28865 test_workload.cc:301] Already present: key already present
*** Check failure stack trace: ***
@ 0x7f74be66b980 (unknown) at ??:0
@ 0x7f74bcb7efb7 gsignal at ??:0
@ 0x7f74bcb80921 abort at ??:0
F20240503 08:13:09.051311 28865 test_workload.cc:301] Already present: key already presentF20240503 08:13:09.063942 28864 test_workload.cc:301] Already present: key already present
*** Check failure stack trace: ***
F20240503 08:13:09.051311 28865 test_workload.cc:301] Already present: key already presentF20240503 08:13:09.063942 28864 test_workload.cc:301] Already present: key already presentF20240503 08:13:09.064047 28867 test_workload.cc:301] Already present: key already present
*** Check failure stack trace: ***
@ 0x7f74bd9e9dcd google::LogMessage::Fail() at ??:0
@ 0x7f74bd9edb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f74bd9e97cc google::LogMessage::Flush() at ??:0
@ 0x7f74bd9eaf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f74befcac26 kudu::TestWorkload::GetNumberOfErrors() at ??:0
@ 0x7f74befcc5bc kudu::TestWorkload::WriteThread() at ??:0
@ 0x7f74bd2066df (unknown) at ??:0
@ 0x7f74be6606db start_thread at ??:0
@ 0x7f74bcc6171f clone at ??:0